yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1269845] [NEW] Failed to get metadata when multiple instances starts up in parallel

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: bsemp <bruno.semperlotti@xxxxxxxxx>
Date: Thu, 16 Jan 2014 15:28:36 -0000
Reply-to: Bug 1269845 <1269845@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

In Grizzly version, when starting multiple VMs at once in horizon (ex: Instance Count = 20), instances can't get keypair from metadata server.
Metadata server returns error 500.

nova-api.log shows these errors:

2014-01-16 15:03:06.841 ERROR nova.api.metadata.handler [req-e1f557bc-ce18-439c-bcc0-267d057e8e90 None None] Failed to get metadata for instance id: d8456517-fda3-4b2c-9bfd-f53987fe299e
2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler Traceback (most recent call last):
2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/api/metadata/handler.py", line 179, in _handle_instance_id_request
2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler     remote_address)
2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/api/metadata/handler.py", line 90, in get_metadata_by_instance_id
2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler     instance_id, address)
2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/api/metadata/base.py", line 417, in get_metadata_by_instance_id
2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler     return InstanceMetadata(instance, address)
2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/api/metadata/base.py", line 143, in __init__
2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler     conductor_api=capi)
2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/network/quantumv2/api.py", line 367, in get_instance_nw_info
2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler     result = self._get_instance_nw_info(context, instance, networks)
2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/network/quantumv2/api.py", line 375, in _get_instance_nw_info
2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler     nw_info = self._build_network_info_model(context, instance, networks)
2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/network/quantumv2/api.py", line 803, in _build_network_info_model
2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler     instance['project_id'])
2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/network/quantumv2/api.py", line 121, in _get_available_networks
2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler     nets += quantum.list_networks(**search_opts).get('networks', [])
2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/quantumclient/v2_0/client.py", line 107, in with_params
2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler     ret = self.function(instance, *args, **kwargs)
2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/quantumclient/v2_0/client.py", line 295, in list_networks
2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler     **_params)
2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/quantumclient/v2_0/client.py", line 999, in list
2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler     for r in self._pagination(collection, path, **params):
2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/quantumclient/v2_0/client.py", line 1012, in _pagination
2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler     res = self.get(path, params=params)
2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/quantumclient/v2_0/client.py", line 985, in get
2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler     headers=headers, params=params)
2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/quantumclient/v2_0/client.py", line 970, in retry_request
2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler     headers=headers, params=params)
2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/quantumclient/v2_0/client.py", line 915, in do_request
2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler     self._handle_fault_response(status_code, replybody)
2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/quantumclient/v2_0/client.py", line 896, in _handle_fault_response
2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler     exception_handler_v20(status_code, des_error_body)
2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/quantumclient/v2_0/client.py", line 87, in exception_handler_v20
2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler     message=message)
2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler QuantumClientException: Second simultaneous read on fileno 16 detected.  Unless you really know what you're doing, make sure that only one greenthread can read any particular socket.  Consider using a pools.Pool. If you do know what you're doing and want to disable this error, call eventlet.debug.hub_prevent_multiple_readers(False)
2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler 
2014-01-16 15:03:06.864 ERROR nova.api.metadata.handler [req-da597f83-1eb6-420a-b1bf-41dcc51208a4 None None] Failed to get metadata for instance id: d8456517-fda3-4b2c-9bfd-f53987fe299e
2014-01-16 15:03:06.864 9378 TRACE nova.api.metadata.handler Traceback (most recent call last):
2014-01-16 15:03:06.864 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/api/metadata/handler.py", line 179, in _handle_instance_id_request
2014-01-16 15:03:06.864 9378 TRACE nova.api.metadata.handler     remote_address)
2014-01-16 15:03:06.864 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/api/metadata/handler.py", line 90, in get_metadata_by_instance_id
2014-01-16 15:03:06.864 9378 TRACE nova.api.metadata.handler     instance_id, address)
2014-01-16 15:03:06.864 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/api/metadata/base.py", line 417, in get_metadata_by_instance_id
2014-01-16 15:03:06.864 9378 TRACE nova.api.metadata.handler     return InstanceMetadata(instance, address)
2014-01-16 15:03:06.864 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/api/metadata/base.py", line 143, in __init__
2014-01-16 15:03:06.864 9378 TRACE nova.api.metadata.handler     conductor_api=capi)
2014-01-16 15:03:06.864 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/network/quantumv2/api.py", line 367, in get_instance_nw_info
2014-01-16 15:03:06.864 9378 TRACE nova.api.metadata.handler     result = self._get_instance_nw_info(context, instance, networks)
2014-01-16 15:03:06.864 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/network/quantumv2/api.py", line 375, in _get_instance_nw_info
2014-01-16 15:03:06.864 9378 TRACE nova.api.metadata.handler     nw_info = self._build_network_info_model(context, instance, networks)
2014-01-16 15:03:06.864 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/network/quantumv2/api.py", line 826, in _build_network_info_model
2014-01-16 15:03:06.864 9378 TRACE nova.api.metadata.handler     client, fixed_ip['ip_address'], port['id'])
2014-01-16 15:03:06.864 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/network/quantumv2/api.py", line 733, in _get_floating_ips_by_fixed_and_port
2014-01-16 15:03:06.864 9378 TRACE nova.api.metadata.handler     port_id=port)
2014-01-16 15:03:06.864 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/quantumclient/v2_0/client.py", line 107, in with_params
2014-01-16 15:03:06.864 9378 TRACE nova.api.metadata.handler     ret = self.function(instance, *args, **kwargs)
2014-01-16 15:03:06.864 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/quantumclient/v2_0/client.py", line 437, in list_floatingips
2014-01-16 15:03:06.864 9378 TRACE nova.api.metadata.handler     **_params)
2014-01-16 15:03:06.864 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/quantumclient/v2_0/client.py", line 1000, in list
2014-01-16 15:03:06.864 9378 TRACE nova.api.metadata.handler     res.extend(r[collection])
2014-01-16 15:03:06.864 9378 TRACE nova.api.metadata.handler KeyError: 'floatingips'
2014-01-16 15:03:06.864 9378 TRACE nova.api.metadata.handler 
2014-01-16 15:03:09.329 ERROR nova.api.metadata.handler [req-ce948675-fd1b-482b-b84b-7826b798b854 None None] Failed to get metadata for instance id: d8456517-fda3-4b2c-9bfd-f53987fe299e
2014-01-16 15:03:09.329 9378 TRACE nova.api.metadata.handler Traceback (most recent call last):
2014-01-16 15:03:09.329 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/api/metadata/handler.py", line 179, in _handle_instance_id_request
2014-01-16 15:03:09.329 9378 TRACE nova.api.metadata.handler     remote_address)
2014-01-16 15:03:09.329 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/api/metadata/handler.py", line 90, in get_metadata_by_instance_id
2014-01-16 15:03:09.329 9378 TRACE nova.api.metadata.handler     instance_id, address)
2014-01-16 15:03:09.329 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/api/metadata/base.py", line 417, in get_metadata_by_instance_id
2014-01-16 15:03:09.329 9378 TRACE nova.api.metadata.handler     return InstanceMetadata(instance, address)
2014-01-16 15:03:09.329 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/api/metadata/base.py", line 143, in __init__
2014-01-16 15:03:09.329 9378 TRACE nova.api.metadata.handler     conductor_api=capi)
2014-01-16 15:03:09.329 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/network/quantumv2/api.py", line 367, in get_instance_nw_info
2014-01-16 15:03:09.329 9378 TRACE nova.api.metadata.handler     result = self._get_instance_nw_info(context, instance, networks)
2014-01-16 15:03:09.329 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/network/quantumv2/api.py", line 375, in _get_instance_nw_info
2014-01-16 15:03:09.329 9378 TRACE nova.api.metadata.handler     nw_info = self._build_network_info_model(context, instance, networks)
2014-01-16 15:03:09.329 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/network/quantumv2/api.py", line 794, in _build_network_info_model
2014-01-16 15:03:09.329 9378 TRACE nova.api.metadata.handler     data = client.list_ports(**search_opts)
2014-01-16 15:03:09.329 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/quantumclient/v2_0/client.py", line 107, in with_params
2014-01-16 15:03:09.329 9378 TRACE nova.api.metadata.handler     ret = self.function(instance, *args, **kwargs)
2014-01-16 15:03:09.329 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/quantumclient/v2_0/client.py", line 258, in list_ports
2014-01-16 15:03:09.329 9378 TRACE nova.api.metadata.handler     **_params)
2014-01-16 15:03:09.329 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/quantumclient/v2_0/client.py", line 1000, in list
2014-01-16 15:03:09.329 9378 TRACE nova.api.metadata.handler     res.extend(r[collection])
2014-01-16 15:03:09.329 9378 TRACE nova.api.metadata.handler KeyError: 'ports'
2014-01-16 15:03:09.329 9378 TRACE nova.api.metadata.handler 
2014-01-16 15:03:09.402 ERROR nova.api.metadata.handler [req-df4760f1-a705-4c0d-a97e-e779f4541dc3 None None] Failed to get metadata for instance id: d8456517-fda3-4b2c-9bfd-f53987fe299e
2014-01-16 15:03:09.402 9378 TRACE nova.api.metadata.handler Traceback (most recent call last):
2014-01-16 15:03:09.402 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/api/metadata/handler.py", line 179, in _handle_instance_id_request
2014-01-16 15:03:09.402 9378 TRACE nova.api.metadata.handler     remote_address)
2014-01-16 15:03:09.402 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/api/metadata/handler.py", line 90, in get_metadata_by_instance_id
2014-01-16 15:03:09.402 9378 TRACE nova.api.metadata.handler     instance_id, address)
2014-01-16 15:03:09.402 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/api/metadata/base.py", line 417, in get_metadata_by_instance_id
2014-01-16 15:03:09.402 9378 TRACE nova.api.metadata.handler     return InstanceMetadata(instance, address)
2014-01-16 15:03:09.402 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/api/metadata/base.py", line 143, in __init__
2014-01-16 15:03:09.402 9378 TRACE nova.api.metadata.handler     conductor_api=capi)
2014-01-16 15:03:09.402 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/network/quantumv2/api.py", line 367, in get_instance_nw_info
2014-01-16 15:03:09.402 9378 TRACE nova.api.metadata.handler     result = self._get_instance_nw_info(context, instance, networks)
2014-01-16 15:03:09.402 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/network/quantumv2/api.py", line 375, in _get_instance_nw_info
2014-01-16 15:03:09.402 9378 TRACE nova.api.metadata.handler     nw_info = self._build_network_info_model(context, instance, networks)
2014-01-16 15:03:09.402 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/network/quantumv2/api.py", line 803, in _build_network_info_model
2014-01-16 15:03:09.402 9378 TRACE nova.api.metadata.handler     instance['project_id'])
2014-01-16 15:03:09.402 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/network/quantumv2/api.py", line 116, in _get_available_networks
2014-01-16 15:03:09.402 9378 TRACE nova.api.metadata.handler     nets = quantum.list_networks(**search_opts).get('networks', [])
2014-01-16 15:03:09.402 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/quantumclient/v2_0/client.py", line 107, in with_params
2014-01-16 15:03:09.402 9378 TRACE nova.api.metadata.handler     ret = self.function(instance, *args, **kwargs)
2014-01-16 15:03:09.402 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/quantumclient/v2_0/client.py", line 295, in list_networks
2014-01-16 15:03:09.402 9378 TRACE nova.api.metadata.handler     **_params)
2014-01-16 15:03:09.402 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/quantumclient/v2_0/client.py", line 1000, in list
2014-01-16 15:03:09.402 9378 TRACE nova.api.metadata.handler     res.extend(r[collection])
2014-01-16 15:03:09.402 9378 TRACE nova.api.metadata.handler KeyError: 'networks'
2014-01-16 15:03:09.402 9378 TRACE nova.api.metadata.handler 
2014-01-16 15:03:11.831 ERROR nova.api.metadata.handler [req-ec5ec4f7-972b-491d-8364-23ad515ee966 None None] Failed to get metadata for instance id: d8456517-fda3-4b2c-9bfd-f53987fe299e
2014-01-16 15:03:11.831 9378 TRACE nova.api.metadata.handler Traceback (most recent call last):
2014-01-16 15:03:11.831 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/api/metadata/handler.py", line 179, in _handle_instance_id_request
2014-01-16 15:03:11.831 9378 TRACE nova.api.metadata.handler     remote_address)
2014-01-16 15:03:11.831 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/api/metadata/handler.py", line 90, in get_metadata_by_instance_id
2014-01-16 15:03:11.831 9378 TRACE nova.api.metadata.handler     instance_id, address)
2014-01-16 15:03:11.831 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/api/metadata/base.py", line 417, in get_metadata_by_instance_id
2014-01-16 15:03:11.831 9378 TRACE nova.api.metadata.handler     return InstanceMetadata(instance, address)
2014-01-16 15:03:11.831 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/api/metadata/base.py", line 143, in __init__
2014-01-16 15:03:11.831 9378 TRACE nova.api.metadata.handler     conductor_api=capi)
2014-01-16 15:03:11.831 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/network/quantumv2/api.py", line 367, in get_instance_nw_info
2014-01-16 15:03:11.831 9378 TRACE nova.api.metadata.handler     result = self._get_instance_nw_info(context, instance, networks)
2014-01-16 15:03:11.831 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/network/quantumv2/api.py", line 375, in _get_instance_nw_info
2014-01-16 15:03:11.831 9378 TRACE nova.api.metadata.handler     nw_info = self._build_network_info_model(context, instance, networks)
2014-01-16 15:03:11.831 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/network/quantumv2/api.py", line 794, in _build_network_info_model
2014-01-16 15:03:11.831 9378 TRACE nova.api.metadata.handler     data = client.list_ports(**search_opts)
2014-01-16 15:03:11.831 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/quantumclient/v2_0/client.py", line 107, in with_params
2014-01-16 15:03:11.831 9378 TRACE nova.api.metadata.handler     ret = self.function(instance, *args, **kwargs)
2014-01-16 15:03:11.831 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/quantumclient/v2_0/client.py", line 258, in list_ports
2014-01-16 15:03:11.831 9378 TRACE nova.api.metadata.handler     **_params)
2014-01-16 15:03:11.831 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/quantumclient/v2_0/client.py", line 1000, in list
2014-01-16 15:03:11.831 9378 TRACE nova.api.metadata.handler     res.extend(r[collection])
2014-01-16 15:03:11.831 9378 TRACE nova.api.metadata.handler KeyError: 'ports'
2014-01-16 15:03:11.831 9378 TRACE nova.api.metadata.handler 
2014-01-16 15:03:11.907 ERROR nova.api.metadata.handler [req-ce797974-fae5-4864-9f31-639bc9db2b70 None None] Failed to get metadata for instance id: d8456517-fda3-4b2c-9bfd-f53987fe299e
2014-01-16 15:03:11.907 9378 TRACE nova.api.metadata.handler Traceback (most recent call last):
2014-01-16 15:03:11.907 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/api/metadata/handler.py", line 179, in _handle_instance_id_request
2014-01-16 15:03:11.907 9378 TRACE nova.api.metadata.handler     remote_address)
2014-01-16 15:03:11.907 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/api/metadata/handler.py", line 90, in get_metadata_by_instance_id
2014-01-16 15:03:11.907 9378 TRACE nova.api.metadata.handler     instance_id, address)
2014-01-16 15:03:11.907 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/api/metadata/base.py", line 417, in get_metadata_by_instance_id
2014-01-16 15:03:11.907 9378 TRACE nova.api.metadata.handler     return InstanceMetadata(instance, address)
2014-01-16 15:03:11.907 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/api/metadata/base.py", line 143, in __init__
2014-01-16 15:03:11.907 9378 TRACE nova.api.metadata.handler     conductor_api=capi)
2014-01-16 15:03:11.907 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/network/quantumv2/api.py", line 367, in get_instance_nw_info
2014-01-16 15:03:11.907 9378 TRACE nova.api.metadata.handler     result = self._get_instance_nw_info(context, instance, networks)
2014-01-16 15:03:11.907 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/network/quantumv2/api.py", line 375, in _get_instance_nw_info
2014-01-16 15:03:11.907 9378 TRACE nova.api.metadata.handler     nw_info = self._build_network_info_model(context, instance, networks)
2014-01-16 15:03:11.907 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/network/quantumv2/api.py", line 803, in _build_network_info_model
2014-01-16 15:03:11.907 9378 TRACE nova.api.metadata.handler     instance['project_id'])
2014-01-16 15:03:11.907 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/network/quantumv2/api.py", line 116, in _get_available_networks
2014-01-16 15:03:11.907 9378 TRACE nova.api.metadata.handler     nets = quantum.list_networks(**search_opts).get('networks', [])
2014-01-16 15:03:11.907 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/quantumclient/v2_0/client.py", line 107, in with_params
2014-01-16 15:03:11.907 9378 TRACE nova.api.metadata.handler     ret = self.function(instance, *args, **kwargs)
2014-01-16 15:03:11.907 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/quantumclient/v2_0/client.py", line 295, in list_networks
2014-01-16 15:03:11.907 9378 TRACE nova.api.metadata.handler     **_params)
2014-01-16 15:03:11.907 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/quantumclient/v2_0/client.py", line 1000, in list
2014-01-16 15:03:11.907 9378 TRACE nova.api.metadata.handler     res.extend(r[collection])
2014-01-16 15:03:11.907 9378 TRACE nova.api.metadata.handler KeyError: 'networks'
2014-01-16 15:03:11.907 9378 TRACE nova.api.metadata.handler 
[...]

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1269845

Title:
  Failed to get metadata when multiple instances starts up in parallel

Status in OpenStack Neutron (virtual network service):
  New

Bug description:
  In Grizzly version, when starting multiple VMs at once in horizon (ex: Instance Count = 20), instances can't get keypair from metadata server.
  Metadata server returns error 500.

  nova-api.log shows these errors:

  2014-01-16 15:03:06.841 ERROR nova.api.metadata.handler [req-e1f557bc-ce18-439c-bcc0-267d057e8e90 None None] Failed to get metadata for instance id: d8456517-fda3-4b2c-9bfd-f53987fe299e
  2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler Traceback (most recent call last):
  2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/api/metadata/handler.py", line 179, in _handle_instance_id_request
  2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler     remote_address)
  2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/api/metadata/handler.py", line 90, in get_metadata_by_instance_id
  2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler     instance_id, address)
  2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/api/metadata/base.py", line 417, in get_metadata_by_instance_id
  2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler     return InstanceMetadata(instance, address)
  2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/api/metadata/base.py", line 143, in __init__
  2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler     conductor_api=capi)
  2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/network/quantumv2/api.py", line 367, in get_instance_nw_info
  2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler     result = self._get_instance_nw_info(context, instance, networks)
  2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/network/quantumv2/api.py", line 375, in _get_instance_nw_info
  2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler     nw_info = self._build_network_info_model(context, instance, networks)
  2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/network/quantumv2/api.py", line 803, in _build_network_info_model
  2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler     instance['project_id'])
  2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/network/quantumv2/api.py", line 121, in _get_available_networks
  2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler     nets += quantum.list_networks(**search_opts).get('networks', [])
  2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/quantumclient/v2_0/client.py", line 107, in with_params
  2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler     ret = self.function(instance, *args, **kwargs)
  2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/quantumclient/v2_0/client.py", line 295, in list_networks
  2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler     **_params)
  2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/quantumclient/v2_0/client.py", line 999, in list
  2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler     for r in self._pagination(collection, path, **params):
  2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/quantumclient/v2_0/client.py", line 1012, in _pagination
  2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler     res = self.get(path, params=params)
  2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/quantumclient/v2_0/client.py", line 985, in get
  2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler     headers=headers, params=params)
  2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/quantumclient/v2_0/client.py", line 970, in retry_request
  2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler     headers=headers, params=params)
  2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/quantumclient/v2_0/client.py", line 915, in do_request
  2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler     self._handle_fault_response(status_code, replybody)
  2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/quantumclient/v2_0/client.py", line 896, in _handle_fault_response
  2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler     exception_handler_v20(status_code, des_error_body)
  2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/quantumclient/v2_0/client.py", line 87, in exception_handler_v20
  2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler     message=message)
  2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler QuantumClientException: Second simultaneous read on fileno 16 detected.  Unless you really know what you're doing, make sure that only one greenthread can read any particular socket.  Consider using a pools.Pool. If you do know what you're doing and want to disable this error, call eventlet.debug.hub_prevent_multiple_readers(False)
  2014-01-16 15:03:06.841 9378 TRACE nova.api.metadata.handler 
  2014-01-16 15:03:06.864 ERROR nova.api.metadata.handler [req-da597f83-1eb6-420a-b1bf-41dcc51208a4 None None] Failed to get metadata for instance id: d8456517-fda3-4b2c-9bfd-f53987fe299e
  2014-01-16 15:03:06.864 9378 TRACE nova.api.metadata.handler Traceback (most recent call last):
  2014-01-16 15:03:06.864 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/api/metadata/handler.py", line 179, in _handle_instance_id_request
  2014-01-16 15:03:06.864 9378 TRACE nova.api.metadata.handler     remote_address)
  2014-01-16 15:03:06.864 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/api/metadata/handler.py", line 90, in get_metadata_by_instance_id
  2014-01-16 15:03:06.864 9378 TRACE nova.api.metadata.handler     instance_id, address)
  2014-01-16 15:03:06.864 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/api/metadata/base.py", line 417, in get_metadata_by_instance_id
  2014-01-16 15:03:06.864 9378 TRACE nova.api.metadata.handler     return InstanceMetadata(instance, address)
  2014-01-16 15:03:06.864 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/api/metadata/base.py", line 143, in __init__
  2014-01-16 15:03:06.864 9378 TRACE nova.api.metadata.handler     conductor_api=capi)
  2014-01-16 15:03:06.864 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/network/quantumv2/api.py", line 367, in get_instance_nw_info
  2014-01-16 15:03:06.864 9378 TRACE nova.api.metadata.handler     result = self._get_instance_nw_info(context, instance, networks)
  2014-01-16 15:03:06.864 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/network/quantumv2/api.py", line 375, in _get_instance_nw_info
  2014-01-16 15:03:06.864 9378 TRACE nova.api.metadata.handler     nw_info = self._build_network_info_model(context, instance, networks)
  2014-01-16 15:03:06.864 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/network/quantumv2/api.py", line 826, in _build_network_info_model
  2014-01-16 15:03:06.864 9378 TRACE nova.api.metadata.handler     client, fixed_ip['ip_address'], port['id'])
  2014-01-16 15:03:06.864 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/network/quantumv2/api.py", line 733, in _get_floating_ips_by_fixed_and_port
  2014-01-16 15:03:06.864 9378 TRACE nova.api.metadata.handler     port_id=port)
  2014-01-16 15:03:06.864 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/quantumclient/v2_0/client.py", line 107, in with_params
  2014-01-16 15:03:06.864 9378 TRACE nova.api.metadata.handler     ret = self.function(instance, *args, **kwargs)
  2014-01-16 15:03:06.864 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/quantumclient/v2_0/client.py", line 437, in list_floatingips
  2014-01-16 15:03:06.864 9378 TRACE nova.api.metadata.handler     **_params)
  2014-01-16 15:03:06.864 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/quantumclient/v2_0/client.py", line 1000, in list
  2014-01-16 15:03:06.864 9378 TRACE nova.api.metadata.handler     res.extend(r[collection])
  2014-01-16 15:03:06.864 9378 TRACE nova.api.metadata.handler KeyError: 'floatingips'
  2014-01-16 15:03:06.864 9378 TRACE nova.api.metadata.handler 
  2014-01-16 15:03:09.329 ERROR nova.api.metadata.handler [req-ce948675-fd1b-482b-b84b-7826b798b854 None None] Failed to get metadata for instance id: d8456517-fda3-4b2c-9bfd-f53987fe299e
  2014-01-16 15:03:09.329 9378 TRACE nova.api.metadata.handler Traceback (most recent call last):
  2014-01-16 15:03:09.329 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/api/metadata/handler.py", line 179, in _handle_instance_id_request
  2014-01-16 15:03:09.329 9378 TRACE nova.api.metadata.handler     remote_address)
  2014-01-16 15:03:09.329 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/api/metadata/handler.py", line 90, in get_metadata_by_instance_id
  2014-01-16 15:03:09.329 9378 TRACE nova.api.metadata.handler     instance_id, address)
  2014-01-16 15:03:09.329 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/api/metadata/base.py", line 417, in get_metadata_by_instance_id
  2014-01-16 15:03:09.329 9378 TRACE nova.api.metadata.handler     return InstanceMetadata(instance, address)
  2014-01-16 15:03:09.329 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/api/metadata/base.py", line 143, in __init__
  2014-01-16 15:03:09.329 9378 TRACE nova.api.metadata.handler     conductor_api=capi)
  2014-01-16 15:03:09.329 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/network/quantumv2/api.py", line 367, in get_instance_nw_info
  2014-01-16 15:03:09.329 9378 TRACE nova.api.metadata.handler     result = self._get_instance_nw_info(context, instance, networks)
  2014-01-16 15:03:09.329 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/network/quantumv2/api.py", line 375, in _get_instance_nw_info
  2014-01-16 15:03:09.329 9378 TRACE nova.api.metadata.handler     nw_info = self._build_network_info_model(context, instance, networks)
  2014-01-16 15:03:09.329 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/network/quantumv2/api.py", line 794, in _build_network_info_model
  2014-01-16 15:03:09.329 9378 TRACE nova.api.metadata.handler     data = client.list_ports(**search_opts)
  2014-01-16 15:03:09.329 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/quantumclient/v2_0/client.py", line 107, in with_params
  2014-01-16 15:03:09.329 9378 TRACE nova.api.metadata.handler     ret = self.function(instance, *args, **kwargs)
  2014-01-16 15:03:09.329 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/quantumclient/v2_0/client.py", line 258, in list_ports
  2014-01-16 15:03:09.329 9378 TRACE nova.api.metadata.handler     **_params)
  2014-01-16 15:03:09.329 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/quantumclient/v2_0/client.py", line 1000, in list
  2014-01-16 15:03:09.329 9378 TRACE nova.api.metadata.handler     res.extend(r[collection])
  2014-01-16 15:03:09.329 9378 TRACE nova.api.metadata.handler KeyError: 'ports'
  2014-01-16 15:03:09.329 9378 TRACE nova.api.metadata.handler 
  2014-01-16 15:03:09.402 ERROR nova.api.metadata.handler [req-df4760f1-a705-4c0d-a97e-e779f4541dc3 None None] Failed to get metadata for instance id: d8456517-fda3-4b2c-9bfd-f53987fe299e
  2014-01-16 15:03:09.402 9378 TRACE nova.api.metadata.handler Traceback (most recent call last):
  2014-01-16 15:03:09.402 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/api/metadata/handler.py", line 179, in _handle_instance_id_request
  2014-01-16 15:03:09.402 9378 TRACE nova.api.metadata.handler     remote_address)
  2014-01-16 15:03:09.402 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/api/metadata/handler.py", line 90, in get_metadata_by_instance_id
  2014-01-16 15:03:09.402 9378 TRACE nova.api.metadata.handler     instance_id, address)
  2014-01-16 15:03:09.402 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/api/metadata/base.py", line 417, in get_metadata_by_instance_id
  2014-01-16 15:03:09.402 9378 TRACE nova.api.metadata.handler     return InstanceMetadata(instance, address)
  2014-01-16 15:03:09.402 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/api/metadata/base.py", line 143, in __init__
  2014-01-16 15:03:09.402 9378 TRACE nova.api.metadata.handler     conductor_api=capi)
  2014-01-16 15:03:09.402 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/network/quantumv2/api.py", line 367, in get_instance_nw_info
  2014-01-16 15:03:09.402 9378 TRACE nova.api.metadata.handler     result = self._get_instance_nw_info(context, instance, networks)
  2014-01-16 15:03:09.402 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/network/quantumv2/api.py", line 375, in _get_instance_nw_info
  2014-01-16 15:03:09.402 9378 TRACE nova.api.metadata.handler     nw_info = self._build_network_info_model(context, instance, networks)
  2014-01-16 15:03:09.402 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/network/quantumv2/api.py", line 803, in _build_network_info_model
  2014-01-16 15:03:09.402 9378 TRACE nova.api.metadata.handler     instance['project_id'])
  2014-01-16 15:03:09.402 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/network/quantumv2/api.py", line 116, in _get_available_networks
  2014-01-16 15:03:09.402 9378 TRACE nova.api.metadata.handler     nets = quantum.list_networks(**search_opts).get('networks', [])
  2014-01-16 15:03:09.402 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/quantumclient/v2_0/client.py", line 107, in with_params
  2014-01-16 15:03:09.402 9378 TRACE nova.api.metadata.handler     ret = self.function(instance, *args, **kwargs)
  2014-01-16 15:03:09.402 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/quantumclient/v2_0/client.py", line 295, in list_networks
  2014-01-16 15:03:09.402 9378 TRACE nova.api.metadata.handler     **_params)
  2014-01-16 15:03:09.402 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/quantumclient/v2_0/client.py", line 1000, in list
  2014-01-16 15:03:09.402 9378 TRACE nova.api.metadata.handler     res.extend(r[collection])
  2014-01-16 15:03:09.402 9378 TRACE nova.api.metadata.handler KeyError: 'networks'
  2014-01-16 15:03:09.402 9378 TRACE nova.api.metadata.handler 
  2014-01-16 15:03:11.831 ERROR nova.api.metadata.handler [req-ec5ec4f7-972b-491d-8364-23ad515ee966 None None] Failed to get metadata for instance id: d8456517-fda3-4b2c-9bfd-f53987fe299e
  2014-01-16 15:03:11.831 9378 TRACE nova.api.metadata.handler Traceback (most recent call last):
  2014-01-16 15:03:11.831 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/api/metadata/handler.py", line 179, in _handle_instance_id_request
  2014-01-16 15:03:11.831 9378 TRACE nova.api.metadata.handler     remote_address)
  2014-01-16 15:03:11.831 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/api/metadata/handler.py", line 90, in get_metadata_by_instance_id
  2014-01-16 15:03:11.831 9378 TRACE nova.api.metadata.handler     instance_id, address)
  2014-01-16 15:03:11.831 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/api/metadata/base.py", line 417, in get_metadata_by_instance_id
  2014-01-16 15:03:11.831 9378 TRACE nova.api.metadata.handler     return InstanceMetadata(instance, address)
  2014-01-16 15:03:11.831 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/api/metadata/base.py", line 143, in __init__
  2014-01-16 15:03:11.831 9378 TRACE nova.api.metadata.handler     conductor_api=capi)
  2014-01-16 15:03:11.831 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/network/quantumv2/api.py", line 367, in get_instance_nw_info
  2014-01-16 15:03:11.831 9378 TRACE nova.api.metadata.handler     result = self._get_instance_nw_info(context, instance, networks)
  2014-01-16 15:03:11.831 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/network/quantumv2/api.py", line 375, in _get_instance_nw_info
  2014-01-16 15:03:11.831 9378 TRACE nova.api.metadata.handler     nw_info = self._build_network_info_model(context, instance, networks)
  2014-01-16 15:03:11.831 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/network/quantumv2/api.py", line 794, in _build_network_info_model
  2014-01-16 15:03:11.831 9378 TRACE nova.api.metadata.handler     data = client.list_ports(**search_opts)
  2014-01-16 15:03:11.831 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/quantumclient/v2_0/client.py", line 107, in with_params
  2014-01-16 15:03:11.831 9378 TRACE nova.api.metadata.handler     ret = self.function(instance, *args, **kwargs)
  2014-01-16 15:03:11.831 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/quantumclient/v2_0/client.py", line 258, in list_ports
  2014-01-16 15:03:11.831 9378 TRACE nova.api.metadata.handler     **_params)
  2014-01-16 15:03:11.831 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/quantumclient/v2_0/client.py", line 1000, in list
  2014-01-16 15:03:11.831 9378 TRACE nova.api.metadata.handler     res.extend(r[collection])
  2014-01-16 15:03:11.831 9378 TRACE nova.api.metadata.handler KeyError: 'ports'
  2014-01-16 15:03:11.831 9378 TRACE nova.api.metadata.handler 
  2014-01-16 15:03:11.907 ERROR nova.api.metadata.handler [req-ce797974-fae5-4864-9f31-639bc9db2b70 None None] Failed to get metadata for instance id: d8456517-fda3-4b2c-9bfd-f53987fe299e
  2014-01-16 15:03:11.907 9378 TRACE nova.api.metadata.handler Traceback (most recent call last):
  2014-01-16 15:03:11.907 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/api/metadata/handler.py", line 179, in _handle_instance_id_request
  2014-01-16 15:03:11.907 9378 TRACE nova.api.metadata.handler     remote_address)
  2014-01-16 15:03:11.907 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/api/metadata/handler.py", line 90, in get_metadata_by_instance_id
  2014-01-16 15:03:11.907 9378 TRACE nova.api.metadata.handler     instance_id, address)
  2014-01-16 15:03:11.907 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/api/metadata/base.py", line 417, in get_metadata_by_instance_id
  2014-01-16 15:03:11.907 9378 TRACE nova.api.metadata.handler     return InstanceMetadata(instance, address)
  2014-01-16 15:03:11.907 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/api/metadata/base.py", line 143, in __init__
  2014-01-16 15:03:11.907 9378 TRACE nova.api.metadata.handler     conductor_api=capi)
  2014-01-16 15:03:11.907 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/network/quantumv2/api.py", line 367, in get_instance_nw_info
  2014-01-16 15:03:11.907 9378 TRACE nova.api.metadata.handler     result = self._get_instance_nw_info(context, instance, networks)
  2014-01-16 15:03:11.907 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/network/quantumv2/api.py", line 375, in _get_instance_nw_info
  2014-01-16 15:03:11.907 9378 TRACE nova.api.metadata.handler     nw_info = self._build_network_info_model(context, instance, networks)
  2014-01-16 15:03:11.907 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/network/quantumv2/api.py", line 803, in _build_network_info_model
  2014-01-16 15:03:11.907 9378 TRACE nova.api.metadata.handler     instance['project_id'])
  2014-01-16 15:03:11.907 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/nova/network/quantumv2/api.py", line 116, in _get_available_networks
  2014-01-16 15:03:11.907 9378 TRACE nova.api.metadata.handler     nets = quantum.list_networks(**search_opts).get('networks', [])
  2014-01-16 15:03:11.907 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/quantumclient/v2_0/client.py", line 107, in with_params
  2014-01-16 15:03:11.907 9378 TRACE nova.api.metadata.handler     ret = self.function(instance, *args, **kwargs)
  2014-01-16 15:03:11.907 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/quantumclient/v2_0/client.py", line 295, in list_networks
  2014-01-16 15:03:11.907 9378 TRACE nova.api.metadata.handler     **_params)
  2014-01-16 15:03:11.907 9378 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.6/site-packages/quantumclient/v2_0/client.py", line 1000, in list
  2014-01-16 15:03:11.907 9378 TRACE nova.api.metadata.handler     res.extend(r[collection])
  2014-01-16 15:03:11.907 9378 TRACE nova.api.metadata.handler KeyError: 'networks'
  2014-01-16 15:03:11.907 9378 TRACE nova.api.metadata.handler 
  [...]

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1269845/+subscriptions
Follow ups

[Bug 1269845] Re: Failed to get metadata when multiple instances starts up in parallel
From: bsemp, 2014-01-20
[Bug 1269845] [NEW] Failed to get metadata when multiple instances starts up in parallel
From: bsemp, 2014-01-16
References

[Bug 1269845] [NEW] Failed to get metadata when multiple instances starts up in parallel
From: bsemp, 2014-01-16