yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1424594] [NEW] 500 error and 2 traces if no free fixed IP left in the neutron network

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: George Shuklin <1424594@xxxxxxxxxxxxxxxxxx>
Date: Mon, 23 Feb 2015 11:23:55 -0000
Reply-to: Bug 1424594 <1424594@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

If nova recieve 404 from neutron due lack of free fixed IPs, it traces
badly and return 500 error to user.

Steps to reproduce:
0. Setup nova & neutron, create network & subnetwork
1. Consume all IP from that network
2. Try to attach interface to that network (nova interface-attach --net-id NET-UUID SERVER-UUID)

Actual behaviour:

ERROR (ClientException): The server has either erred or is incapable of
performing the requested operation. (HTTP 500) (Request-ID: req-
994444ec-6a69-428d-9c16-c58d685553dd)

... and traces (see below)

Expected behaviour:

Proper complain about lack of IP (NoMoreFixedIps) and proper HTTP error
code.

Traces (nova-api):

nova.api.openstack.wsgi[26783]: DEBUG Action: 'create', calling method: <bound method InterfaceAttachmentController.create of <nova.api.openstack.compute.contrib.attach_interfaces.InterfaceAttachmentController object at 0x7f098e01b390>>, body: {"interfaceAttachment": {"net_id": "f3f2802a-c2a1-4d8b-9f43-cf24d0dc8233"}} [req-57f4e821-a968-48cd-8358-f73fa16b4ff7 4aac5cb61b1741b2a32067619555ecc1 78ea359977584bcc9feceef2553dbe57] _process_stack /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:934
nova.api.openstack.compute.contrib.attach_interfaces[26783]: AUDIT [instance: 5f1e84cb-1766-45e1-899b-9de1e535309b] Attach interface [req-57f4e821-a968-48cd-8358-f73fa16b4ff7 4aac5cb61b1741b2a32067619555ecc1 78ea359977584bcc9feceef2553dbe57]
nova.api.openstack[26783]: ERROR Caught error: Zero fixed ips available.
Traceback (most recent call last):

  File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
    incoming.message))

  File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)

  File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
    result = getattr(endpoint, method)(ctxt, **new_args)

  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 419, in decorated_function
    return function(self, context, *args, **kwargs)

  File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped
    payload)

  File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 82, in __exit__
    six.reraise(self.type_, self.value, self.tb)

  File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 71, in wrapped
    return f(self, context, *args, **kw)

  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 303, in decorated_function
    pass

  File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 82, in __exit__
    six.reraise(self.type_, self.value, self.tb)

  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 289, in decorated_function
    return function(self, context, *args, **kwargs)

  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 331, in decorated_function
    kwargs['instance'], e, sys.exc_info())

  File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 82, in __exit__
    six.reraise(self.type_, self.value, self.tb)

  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 319, in decorated_function
    return function(self, context, *args, **kwargs)

  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4787, in attach_interface
    context, instance, port_id, network_id, requested_ip)

  File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 569, in allocate_port_for_instance
    requested_networks=requested_networks)

  File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 443, in allocate_for_instance
    self._delete_ports(neutron, instance, created_port_ids)

  File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 82, in __exit__
    six.reraise(self.type_, self.value, self.tb)

  File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 423, in allocate_for_instance
    security_group_ids, available_macs, dhcp_opts)

  File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 226, in _create_port
    raise exception.NoMoreFixedIps()

NoMoreFixedIps: Zero fixed ips available.
 [req-57f4e821-a968-48cd-8358-f73fa16b4ff7 4aac5cb61b1741b2a32067619555ecc1 78ea359977584bcc9feceef2553dbe57]
nova.api.openstack[26783]: TRACE Traceback (most recent call last):
nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 124, in __call__
nova.api.openstack[26783]: TRACE     return req.get_response(self.application)
nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
nova.api.openstack[26783]: TRACE     application, catch_exc_info=False)
nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
nova.api.openstack[26783]: TRACE     app_iter = application(self.environ, start_response)
nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
nova.api.openstack[26783]: TRACE     return resp(environ, start_response)
nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py", line 661, in __call__
nova.api.openstack[26783]: TRACE     return self._app(env, start_response)
nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
nova.api.openstack[26783]: TRACE     return resp(environ, start_response)
nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
nova.api.openstack[26783]: TRACE     return resp(environ, start_response)
nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
nova.api.openstack[26783]: TRACE     response = self.app(environ, start_response)
nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
nova.api.openstack[26783]: TRACE     return resp(environ, start_response)
nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
nova.api.openstack[26783]: TRACE     resp = self.call_func(req, *args, **self.kwargs)
nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
nova.api.openstack[26783]: TRACE     return self.func(req, *args, **kwargs)
nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 910, in __call__
nova.api.openstack[26783]: TRACE     content_type, body, accept)
nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 975, in _process_stack
nova.api.openstack[26783]: TRACE     action_result = self.dispatch(meth, request, action_args)
nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 1059, in dispatch
nova.api.openstack[26783]: TRACE     return method(req=request, **action_args)
nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/attach_interfaces.py", line 109, in create
nova.api.openstack[26783]: TRACE     instance, network_id, port_id, req_ip)
nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 220, in wrapped
nova.api.openstack[26783]: TRACE     return func(self, context, target, *args, **kwargs)
nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 210, in inner
nova.api.openstack[26783]: TRACE     return function(self, context, instance, *args, **kwargs)
nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 191, in inner
nova.api.openstack[26783]: TRACE     return f(self, context, instance, *args, **kw)
nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 3010, in attach_interface
nova.api.openstack[26783]: TRACE     requested_ip=requested_ip)
nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/compute/rpcapi.py", line 335, in attach_interface
nova.api.openstack[26783]: TRACE     port_id=port_id, requested_ip=requested_ip)
nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 152, in call
nova.api.openstack[26783]: TRACE     retry=self.retry)
nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 90, in _send
nova.api.openstack[26783]: TRACE     timeout=timeout, retry=retry)
nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 408, in send
nova.api.openstack[26783]: TRACE     retry=retry)
nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 399, in _send
nova.api.openstack[26783]: TRACE     raise result
nova.api.openstack[26783]: TRACE NoMoreFixedIps_Remote: Zero fixed ips available.
nova.api.openstack[26783]: TRACE Traceback (most recent call last):
nova.api.openstack[26783]: TRACE 
nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
nova.api.openstack[26783]: TRACE     incoming.message))
nova.api.openstack[26783]: TRACE 
nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
nova.api.openstack[26783]: TRACE     return self._do_dispatch(endpoint, method, ctxt, args)
nova.api.openstack[26783]: TRACE 
nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
nova.api.openstack[26783]: TRACE     result = getattr(endpoint, method)(ctxt, **new_args)
nova.api.openstack[26783]: TRACE 
nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 419, in decorated_function
nova.api.openstack[26783]: TRACE     return function(self, context, *args, **kwargs)
nova.api.openstack[26783]: TRACE 
nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped
nova.api.openstack[26783]: TRACE     payload)
nova.api.openstack[26783]: TRACE 
nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 82, in __exit__
nova.api.openstack[26783]: TRACE     six.reraise(self.type_, self.value, self.tb)
nova.api.openstack[26783]: TRACE 
nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 71, in wrapped
nova.api.openstack[26783]: TRACE     return f(self, context, *args, **kw)
nova.api.openstack[26783]: TRACE 
nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 303, in decorated_function
nova.api.openstack[26783]: TRACE     pass
nova.api.openstack[26783]: TRACE 
nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 82, in __exit__
nova.api.openstack[26783]: TRACE     six.reraise(self.type_, self.value, self.tb)
nova.api.openstack[26783]: TRACE 
nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 289, in decorated_function
nova.api.openstack[26783]: TRACE     return function(self, context, *args, **kwargs)
nova.api.openstack[26783]: TRACE 
nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 331, in decorated_function
nova.api.openstack[26783]: TRACE     kwargs['instance'], e, sys.exc_info())
nova.api.openstack[26783]: TRACE 
nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 82, in __exit__
nova.api.openstack[26783]: TRACE     six.reraise(self.type_, self.value, self.tb)
nova.api.openstack[26783]: TRACE 
nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 319, in decorated_function
nova.api.openstack[26783]: TRACE     return function(self, context, *args, **kwargs)
nova.api.openstack[26783]: TRACE 
nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4787, in attach_interface
nova.api.openstack[26783]: TRACE     context, instance, port_id, network_id, requested_ip)
nova.api.openstack[26783]: TRACE 
nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 569, in allocate_port_for_instance
nova.api.openstack[26783]: TRACE     requested_networks=requested_networks)
nova.api.openstack[26783]: TRACE 
nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 443, in allocate_for_instance
nova.api.openstack[26783]: TRACE     self._delete_ports(neutron, instance, created_port_ids)
nova.api.openstack[26783]: TRACE 
nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 82, in __exit__
nova.api.openstack[26783]: TRACE     six.reraise(self.type_, self.value, self.tb)
nova.api.openstack[26783]: TRACE 
nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 423, in allocate_for_instance
nova.api.openstack[26783]: TRACE     security_group_ids, available_macs, dhcp_opts)
nova.api.openstack[26783]: TRACE 
nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 226, in _create_port
nova.api.openstack[26783]: TRACE     raise exception.NoMoreFixedIps()
nova.api.openstack[26783]: TRACE 
nova.api.openstack[26783]: TRACE NoMoreFixedIps: Zero fixed ips available.
nova.api.openstack[26783]: TRACE 
nova.api.openstack[26783]: TRACE 
nova.api.openstack[26783]: INFO http://isokolov1.compute.dev.internal:8774/v2/78ea359977584bcc9feceef2553dbe57/servers/5f1e84cb-1766-45e1-899b-9de1e535309b/os-interface returned with HTTP 500 [req-57f4e821-a968-48cd-8358-f73fa16b4ff7 4aac5cb61b1741b2a32067619555ecc1 78ea359977584bcc9feceef2553dbe57]
nova.api.openstack.wsgi[26783]: DEBUG Returning 500 to user: The server has either erred or is incapable of performing the requested operation. [req-57f4e821-a968-48cd-8358-f73fa16b4ff7 4aac5cb61b1741b2a32067619555ecc1 78ea359977584bcc9feceef2553dbe57] __call__ /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:1201
nova.osapi_compute.wsgi.server[26783]: INFO 192.168.32.70 "POST /v2/78ea359977584bcc9feceef2553dbe57/servers/5f1e84cb-1766-45e1-899b-9de1e535309b/os-interface HTTP/1.1" status: 500 len: 335 time: 0.5942502 [req-57f4e821-a968-48cd-8358-f73fa16b4ff7 4aac5cb61b1741b2a32067619555ecc1 78ea359977584bcc9feceef2553dbe57]
nova.api.openstack.wsgi[26782]: DEBUG Calling method '<bound method Controller.detail of <nova.api.openstack.compute.servers.Controller object at 0x7f098ebe6610>>' [req-f51d4aa9-0b6b-4b84-a247-04e650055589 44cc5bff123b4fe0b06f6ac90418756d b80de3539fd2406ea4c0d7c827c4fdd7] _process_stack /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:937
nova.compute.api[26782]: DEBUG Searching by: {'deleted': False} [req-f51d4aa9-0b6b-4b84-a247-04e650055589 44cc5bff123b4fe0b06f6ac90418756d b80de3539fd2406ea4c0d7c827c4fdd7] get_all /usr/lib/python2.7/dist-packages/nova/compute/api.py:1920
neutronclient.client[26782]: DEBUG 
REQ: curl -i http://neutron-server.p.isokolov1.compute.serverscom.dev:29696/v2.0/ports.json?device_id=3ede6afc-7695-414c-bff0-4b807fe4644c&device_id=bf4abf26-b937-4f2d-8c29-6e69ad4be28c&device_id=375cd096-8022-4721-bcc9-5d65f1d21865&device_id=e8378206-ce54-45c3-976c-d3a38a8c619d&device_id=5f1e84cb-1766-45e1-899b-9de1e535309b -X GET -H "X-Auth-Token: bcc1f6eff61a434d9165794c082e7102" -H "User-Agent: python-neutronclient"
 [req-f51d4aa9-0b6b-4b84-a247-04e650055589 44cc5bff123b4fe0b06f6ac90418756d b80de3539fd2406ea4c0d7c827c4fdd7] http_log_req /usr/lib/python2.7/dist-packages/neutronclient/common/utils.py:138
neutronclient.client[26782]: DEBUG RESP:200 CaseInsensitiveDict({'date': 'Mon, 23 Feb 2015 11:07:39 GMT', 'content-length': '2182', 'content-type': 'application/json; charset=UTF-8', 'x-openstack-request-id': 'req-809c50d1-d35c-4b0e-88c6-f9730d540216'}) {"ports": [{"status": "ACTIVE", "binding:host_id": "compute1", "allowed_address_pairs": [], "extra_dhcp_opts": [], "device_owner": "compute:nova", "binding:profile": {}, "fixed_ips": [{"subnet_id": "a4180ba0-6681-4e07-8cc8-0617cccfd12e", "ip_address": "191.168.1.10"}], "id": "11f8456d-41f9-46e7-b6c8-5f6becc59a07", "security_groups": ["0700a947-91aa-49a2-8cd6-c1e2abd8d8c4"], "device_id": "375cd096-8022-4721-bcc9-5d65f1d21865", "name": "", "admin_state_up": true, "network_id": "7ad70c6b-82f6-4723-be8e-d96047755f8e", "tenant_id": "b6749c7410594a01ae857f78d24884e3", "binding:vif_details": {"port_filter": true, "ovs_hybrid_plug": true}, "binding:vnic_type": "normal", "binding:vif_type": "ovs", "mac_address": "fa:16:3e:9a:09:e0"}, {"status": "ACTIVE", "binding:host_id": "compute1", "allowed_address_pairs": [], "extra_dhcp_opts": [], "device_owner": "compute:nova", "binding:profile": {}, "fixed_ips": [{"subnet_id": "85412a00-c7da-472a-8cfe-3f66f911263a", "ip_address": "192.168.1.1"}], "id": "3e66374d-54d2-4efc-be86-0c2359c90c7c", "security_groups": ["0700a947-91aa-49a2-8cd6-c1e2abd8d8c4"], "device_id": "3ede6afc-7695-414c-bff0-4b807fe4644c", "name": "", "admin_state_up": true, "network_id": "674ffba2-6393-4a77-85d2-cceab7960f28", "tenant_id": "b6749c7410594a01ae857f78d24884e3", "binding:vif_details": {"port_filter": true, "ovs_hybrid_plug": true}, "binding:vnic_type": "normal", "binding:vif_type": "ovs", "mac_address": "fa:16:3e:8c:ef:3a"}, {"status": "ACTIVE", "binding:host_id": "compute1", "allowed_address_pairs": [], "extra_dhcp_opts": [], "device_owner": "compute:nova", "binding:profile": {}, "fixed_ips": [{"subnet_id": "50ebc948-752c-48a4-9825-77a43deb1363", "ip_address": "193.168.1.10"}], "id": "a9deee70-6a99-4d4f-8e0c-9ba05a0d61f3", "security_groups": ["0700a947-91aa-49a2-8cd6-c1e2abd8d8c4"], "device_id": "bf4abf26-b937-4f2d-8c29-6e69ad4be28c", "name": "", "admin_state_up": true, "network_id": "f3f2802a-c2a1-4d8b-9f43-cf24d0dc8233", "tenant_id": "b6749c7410594a01ae857f78d24884e3", "binding:vif_details": {"port_filter": true, "ovs_hybrid_plug": true}, "binding:vnic_type": "normal", "binding:vif_type": "ovs", "mac_address": "fa:16:3e:a5:10:3e"}]}
 [req-f51d4aa9-0b6b-4b84-a247-04e650055589 44cc5bff123b4fe0b06f6ac90418756d b80de3539fd2406ea4c0d7c827c4fdd7] http_log_resp /usr/lib/python2.7/dist-packages/neutronclient/common/utils.py:147
neutronclient.client[26782]: DEBUG 
REQ: curl -i http://neutron-server.p.isokolov1.compute.serverscom.dev:29696/v2.0/security-groups.json?id=0700a947-91aa-49a2-8cd6-c1e2abd8d8c4 -X GET -H "X-Auth-Token: bcc1f6eff61a434d9165794c082e7102" -H "User-Agent: python-neutronclient"
 [req-f51d4aa9-0b6b-4b84-a247-04e650055589 44cc5bff123b4fe0b06f6ac90418756d b80de3539fd2406ea4c0d7c827c4fdd7] http_log_req /usr/lib/python2.7/dist-packages/neutronclient/common/utils.py:138
neutronclient.client[26782]: DEBUG RESP:200 CaseInsensitiveDict({'date': 'Mon, 23 Feb 2015 11:07:39 GMT', 'content-length': '1535', 'content-type': 'application/json; charset=UTF-8', 'x-openstack-request-id': 'req-d048f970-115c-4ed1-a7d4-5c95d7a4642d'}) {"security_groups": [{"tenant_id": "b6749c7410594a01ae857f78d24884e3", "description": "default", "id": "0700a947-91aa-49a2-8cd6-c1e2abd8d8c4", "security_group_rules": [{"remote_group_id": "0700a947-91aa-49a2-8cd6-c1e2abd8d8c4", "direction": "ingress", "remote_ip_prefix": null, "protocol": null, "ethertype": "IPv4", "tenant_id": "b6749c7410594a01ae857f78d24884e3", "port_range_max": null, "port_range_min": null, "id": "259b62a6-7283-41db-8653-6578418aec00", "security_group_id": "0700a947-91aa-49a2-8cd6-c1e2abd8d8c4"}, {"remote_group_id": "0700a947-91aa-49a2-8cd6-c1e2abd8d8c4", "direction": "ingress", "remote_ip_prefix": null, "protocol": null, "ethertype": "IPv6", "tenant_id": "b6749c7410594a01ae857f78d24884e3", "port_range_max": null, "port_range_min": null, "id": "7b36cbe8-f616-4997-90b1-d0a684fff7f1", "security_group_id": "0700a947-91aa-49a2-8cd6-c1e2abd8d8c4"}, {"remote_group_id": null, "direction": "egress", "remote_ip_prefix": null, "protocol": null, "ethertype": "IPv4", "tenant_id": "b6749c7410594a01ae857f78d24884e3", "port_range_max": null, "port_range_min": null, "id": "9e9aa8e9-5bf8-427d-8c65-27022d39355f", "security_group_id": "0700a947-91aa-49a2-8cd6-c1e2abd8d8c4"}, {"remote_group_id": null, "direction": "egress", "remote_ip_prefix": null, "protocol": null, "ethertype": "IPv6", "tenant_id": "b6749c7410594a01ae857f78d24884e3", "port_range_max": null, "port_range_min": null, "id": "dbd6a976-0cf1-48d3-a437-6bf4a5dbb0ab", "security_group_id": "0700a947-91aa-49a2-8cd6-c1e2abd8d8c4"}], "name": "default"}]}
 [req-f51d4aa9-0b6b-4b84-a247-04e650055589 44cc5bff123b4fe0b06f6ac90418756d b80de3539fd2406ea4c0d7c827c4fdd7] http_log_resp /usr/lib/python2.7/dist-packages/neutronclient/common/utils.py:147
nova.osapi_compute.wsgi.server[26782]: INFO 192.168.32.70 "GET /v2/b80de3539fd2406ea4c0d7c827c4fdd7/servers/detail?all_tenants=True HTTP/1.1" status: 200 len: 9633 time: 0.3986661 [req-f51d4aa9-0b6b-4b84-a247-04e650055589 44cc5bff123b4fe0b06f6ac90418756d b80de3539fd2406ea4c0d7c827c4fdd7]
nova.api.openstack.wsgi[26781]: DEBUG Calling method '<bound method FloatingIPController.index of <nova.api.openstack.compute.contrib.floating_ips.FloatingIPController object at 0x7f098e02d890>>' [req-0383f71c-5767-482b-9ee4-53c0b4ff6806 44cc5bff123b4fe0b06f6ac90418756d b80de3539fd2406ea4c0d7c827c4fdd7] _process_stack /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:937
neutronclient.client[26781]: DEBUG 
REQ: curl -i http://neutron-server.p.isokolov1.compute.serverscom.dev:29696/v2.0/floatingips.json?tenant_id=b80de3539fd2406ea4c0d7c827c4fdd7 -X GET -H "X-Auth-Token: e775fc86f0f4418999241bfc779bc397" -H "User-Agent: python-neutronclient"
 [req-0383f71c-5767-482b-9ee4-53c0b4ff6806 44cc5bff123b4fe0b06f6ac90418756d b80de3539fd2406ea4c0d7c827c4fdd7] http_log_req /usr/lib/python2.7/dist-packages/neutronclient/common/utils.py:138
neutronclient.client[26781]: DEBUG RESP:404 CaseInsensitiveDict({'date': 'Mon, 23 Feb 2015 11:07:40 GMT', 'content-length': '52', 'content-type': 'text/plain; charset=UTF-8', 'x-openstack-request-id': 'req-79af3c6f-c2d6-4899-af28-f31313a82dbe'}) 404 Not Found

The resource could not be found.

   
 [req-0383f71c-5767-482b-9ee4-53c0b4ff6806 44cc5bff123b4fe0b06f6ac90418756d b80de3539fd2406ea4c0d7c827c4fdd7] http_log_resp /usr/lib/python2.7/dist-packages/neutronclient/common/utils.py:147
neutronclient.v2_0.client[26781]: DEBUG Error message: 404 Not Found

The resource could not be found.

    [req-0383f71c-5767-482b-9ee4-53c0b4ff6806 44cc5bff123b4fe0b06f6ac90418756d b80de3539fd2406ea4c0d7c827c4fdd7] _handle_fault_response /usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py:1208
nova.api.openstack[26781]: ERROR Caught error: 404 Not Found

The resource could not be found.

    [req-0383f71c-5767-482b-9ee4-53c0b4ff6806 44cc5bff123b4fe0b06f6ac90418756d b80de3539fd2406ea4c0d7c827c4fdd7]
nova.api.openstack[26781]: TRACE Traceback (most recent call last):
nova.api.openstack[26781]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 124, in __call__
nova.api.openstack[26781]: TRACE     return req.get_response(self.application)
nova.api.openstack[26781]: TRACE   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
nova.api.openstack[26781]: TRACE     application, catch_exc_info=False)
nova.api.openstack[26781]: TRACE   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
nova.api.openstack[26781]: TRACE     app_iter = application(self.environ, start_response)
nova.api.openstack[26781]: TRACE   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
nova.api.openstack[26781]: TRACE     return resp(environ, start_response)
nova.api.openstack[26781]: TRACE   File "/usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py", line 661, in __call__
nova.api.openstack[26781]: TRACE     return self._app(env, start_response)
nova.api.openstack[26781]: TRACE   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
nova.api.openstack[26781]: TRACE     return resp(environ, start_response)
nova.api.openstack[26781]: TRACE   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
nova.api.openstack[26781]: TRACE     return resp(environ, start_response)
nova.api.openstack[26781]: TRACE   File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
nova.api.openstack[26781]: TRACE     response = self.app(environ, start_response)
nova.api.openstack[26781]: TRACE   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
nova.api.openstack[26781]: TRACE     return resp(environ, start_response)
nova.api.openstack[26781]: TRACE   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
nova.api.openstack[26781]: TRACE     resp = self.call_func(req, *args, **self.kwargs)
nova.api.openstack[26781]: TRACE   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
nova.api.openstack[26781]: TRACE     return self.func(req, *args, **kwargs)
nova.api.openstack[26781]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 910, in __call__
nova.api.openstack[26781]: TRACE     content_type, body, accept)
nova.api.openstack[26781]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 975, in _process_stack
nova.api.openstack[26781]: TRACE     action_result = self.dispatch(meth, request, action_args)
nova.api.openstack[26781]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 1059, in dispatch
nova.api.openstack[26781]: TRACE     return method(req=request, **action_args)
nova.api.openstack[26781]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/floating_ips.py", line 135, in index
nova.api.openstack[26781]: TRACE     floating_ips = self.network_api.get_floating_ips_by_project(context)
nova.api.openstack[26781]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 1096, in get_floating_ips_by_project
nova.api.openstack[26781]: TRACE     fips = client.list_floatingips(tenant_id=project_id)['floatingips']
nova.api.openstack[26781]: TRACE   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 99, in with_params
nova.api.openstack[26781]: TRACE     ret = self.function(instance, *args, **kwargs)
nova.api.openstack[26781]: TRACE   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 444, in list_floatingips
nova.api.openstack[26781]: TRACE     **_params)
nova.api.openstack[26781]: TRACE   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1335, in list
nova.api.openstack[26781]: TRACE     for r in self._pagination(collection, path, **params):
nova.api.openstack[26781]: TRACE   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1348, in _pagination
nova.api.openstack[26781]: TRACE     res = self.get(path, params=params)
nova.api.openstack[26781]: TRACE   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1321, in get
nova.api.openstack[26781]: TRACE     headers=headers, params=params)
nova.api.openstack[26781]: TRACE   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1298, in retry_request
nova.api.openstack[26781]: TRACE     headers=headers, params=params)
nova.api.openstack[26781]: TRACE   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1252, in do_request
nova.api.openstack[26781]: TRACE     self._handle_fault_response(status_code, replybody)
nova.api.openstack[26781]: TRACE   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1217, in _handle_fault_response
nova.api.openstack[26781]: TRACE     exception_handler_v20(status_code, des_error_body)
nova.api.openstack[26781]: TRACE   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 80, in exception_handler_v20
nova.api.openstack[26781]: TRACE     message=message)
nova.api.openstack[26781]: TRACE NeutronClientException: 404 Not Found
nova.api.openstack[26781]: TRACE 
nova.api.openstack[26781]: TRACE The resource could not be found.
nova.api.openstack[26781]: TRACE 
nova.api.openstack[26781]: TRACE    
nova.api.openstack[26781]: TRACE 
nova.api.openstack[26781]: INFO http://isokolov1.compute.dev.internal:8774/v2/b80de3539fd2406ea4c0d7c827c4fdd7/os-floating-ips returned with HTTP 500 [req-0383f71c-5767-482b-9ee4-53c0b4ff6806 44cc5bff123b4fe0b06f6ac90418756d b80de3539fd2406ea4c0d7c827c4fdd7]
nova.api.openstack.wsgi[26781]: DEBUG Returning 500 to user: The server has either erred or is incapable of performing the requested operation. [req-0383f71c-5767-482b-9ee4-53c0b4ff6806 44cc5bff123b4fe0b06f6ac90418756d b80de3539fd2406ea4c0d7c827c4fdd7] __call__ /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:1201

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1424594

Title:
  500 error and 2 traces if no free fixed IP left in the neutron network

Status in OpenStack Compute (Nova):
  New

Bug description:
  If nova recieve 404 from neutron due lack of free fixed IPs, it traces
  badly and return 500 error to user.

  Steps to reproduce:
  0. Setup nova & neutron, create network & subnetwork
  1. Consume all IP from that network
  2. Try to attach interface to that network (nova interface-attach --net-id NET-UUID SERVER-UUID)

  Actual behaviour:

  ERROR (ClientException): The server has either erred or is incapable
  of performing the requested operation. (HTTP 500) (Request-ID: req-
  994444ec-6a69-428d-9c16-c58d685553dd)

  ... and traces (see below)

  Expected behaviour:

  Proper complain about lack of IP (NoMoreFixedIps) and proper HTTP
  error code.

  Traces (nova-api):

  nova.api.openstack.wsgi[26783]: DEBUG Action: 'create', calling method: <bound method InterfaceAttachmentController.create of <nova.api.openstack.compute.contrib.attach_interfaces.InterfaceAttachmentController object at 0x7f098e01b390>>, body: {"interfaceAttachment": {"net_id": "f3f2802a-c2a1-4d8b-9f43-cf24d0dc8233"}} [req-57f4e821-a968-48cd-8358-f73fa16b4ff7 4aac5cb61b1741b2a32067619555ecc1 78ea359977584bcc9feceef2553dbe57] _process_stack /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:934
  nova.api.openstack.compute.contrib.attach_interfaces[26783]: AUDIT [instance: 5f1e84cb-1766-45e1-899b-9de1e535309b] Attach interface [req-57f4e821-a968-48cd-8358-f73fa16b4ff7 4aac5cb61b1741b2a32067619555ecc1 78ea359977584bcc9feceef2553dbe57]
  nova.api.openstack[26783]: ERROR Caught error: Zero fixed ips available.
  Traceback (most recent call last):

    File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
      incoming.message))

    File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
      return self._do_dispatch(endpoint, method, ctxt, args)

    File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
      result = getattr(endpoint, method)(ctxt, **new_args)

    File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 419, in decorated_function
      return function(self, context, *args, **kwargs)

    File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped
      payload)

    File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 82, in __exit__
      six.reraise(self.type_, self.value, self.tb)

    File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 71, in wrapped
      return f(self, context, *args, **kw)

    File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 303, in decorated_function
      pass

    File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 82, in __exit__
      six.reraise(self.type_, self.value, self.tb)

    File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 289, in decorated_function
      return function(self, context, *args, **kwargs)

    File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 331, in decorated_function
      kwargs['instance'], e, sys.exc_info())

    File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 82, in __exit__
      six.reraise(self.type_, self.value, self.tb)

    File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 319, in decorated_function
      return function(self, context, *args, **kwargs)

    File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4787, in attach_interface
      context, instance, port_id, network_id, requested_ip)

    File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 569, in allocate_port_for_instance
      requested_networks=requested_networks)

    File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 443, in allocate_for_instance
      self._delete_ports(neutron, instance, created_port_ids)

    File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 82, in __exit__
      six.reraise(self.type_, self.value, self.tb)

    File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 423, in allocate_for_instance
      security_group_ids, available_macs, dhcp_opts)

    File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 226, in _create_port
      raise exception.NoMoreFixedIps()

  NoMoreFixedIps: Zero fixed ips available.
   [req-57f4e821-a968-48cd-8358-f73fa16b4ff7 4aac5cb61b1741b2a32067619555ecc1 78ea359977584bcc9feceef2553dbe57]
  nova.api.openstack[26783]: TRACE Traceback (most recent call last):
  nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 124, in __call__
  nova.api.openstack[26783]: TRACE     return req.get_response(self.application)
  nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
  nova.api.openstack[26783]: TRACE     application, catch_exc_info=False)
  nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
  nova.api.openstack[26783]: TRACE     app_iter = application(self.environ, start_response)
  nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  nova.api.openstack[26783]: TRACE     return resp(environ, start_response)
  nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py", line 661, in __call__
  nova.api.openstack[26783]: TRACE     return self._app(env, start_response)
  nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  nova.api.openstack[26783]: TRACE     return resp(environ, start_response)
  nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  nova.api.openstack[26783]: TRACE     return resp(environ, start_response)
  nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
  nova.api.openstack[26783]: TRACE     response = self.app(environ, start_response)
  nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  nova.api.openstack[26783]: TRACE     return resp(environ, start_response)
  nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
  nova.api.openstack[26783]: TRACE     resp = self.call_func(req, *args, **self.kwargs)
  nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
  nova.api.openstack[26783]: TRACE     return self.func(req, *args, **kwargs)
  nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 910, in __call__
  nova.api.openstack[26783]: TRACE     content_type, body, accept)
  nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 975, in _process_stack
  nova.api.openstack[26783]: TRACE     action_result = self.dispatch(meth, request, action_args)
  nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 1059, in dispatch
  nova.api.openstack[26783]: TRACE     return method(req=request, **action_args)
  nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/attach_interfaces.py", line 109, in create
  nova.api.openstack[26783]: TRACE     instance, network_id, port_id, req_ip)
  nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 220, in wrapped
  nova.api.openstack[26783]: TRACE     return func(self, context, target, *args, **kwargs)
  nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 210, in inner
  nova.api.openstack[26783]: TRACE     return function(self, context, instance, *args, **kwargs)
  nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 191, in inner
  nova.api.openstack[26783]: TRACE     return f(self, context, instance, *args, **kw)
  nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 3010, in attach_interface
  nova.api.openstack[26783]: TRACE     requested_ip=requested_ip)
  nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/compute/rpcapi.py", line 335, in attach_interface
  nova.api.openstack[26783]: TRACE     port_id=port_id, requested_ip=requested_ip)
  nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 152, in call
  nova.api.openstack[26783]: TRACE     retry=self.retry)
  nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 90, in _send
  nova.api.openstack[26783]: TRACE     timeout=timeout, retry=retry)
  nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 408, in send
  nova.api.openstack[26783]: TRACE     retry=retry)
  nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 399, in _send
  nova.api.openstack[26783]: TRACE     raise result
  nova.api.openstack[26783]: TRACE NoMoreFixedIps_Remote: Zero fixed ips available.
  nova.api.openstack[26783]: TRACE Traceback (most recent call last):
  nova.api.openstack[26783]: TRACE 
  nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
  nova.api.openstack[26783]: TRACE     incoming.message))
  nova.api.openstack[26783]: TRACE 
  nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
  nova.api.openstack[26783]: TRACE     return self._do_dispatch(endpoint, method, ctxt, args)
  nova.api.openstack[26783]: TRACE 
  nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
  nova.api.openstack[26783]: TRACE     result = getattr(endpoint, method)(ctxt, **new_args)
  nova.api.openstack[26783]: TRACE 
  nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 419, in decorated_function
  nova.api.openstack[26783]: TRACE     return function(self, context, *args, **kwargs)
  nova.api.openstack[26783]: TRACE 
  nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped
  nova.api.openstack[26783]: TRACE     payload)
  nova.api.openstack[26783]: TRACE 
  nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 82, in __exit__
  nova.api.openstack[26783]: TRACE     six.reraise(self.type_, self.value, self.tb)
  nova.api.openstack[26783]: TRACE 
  nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 71, in wrapped
  nova.api.openstack[26783]: TRACE     return f(self, context, *args, **kw)
  nova.api.openstack[26783]: TRACE 
  nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 303, in decorated_function
  nova.api.openstack[26783]: TRACE     pass
  nova.api.openstack[26783]: TRACE 
  nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 82, in __exit__
  nova.api.openstack[26783]: TRACE     six.reraise(self.type_, self.value, self.tb)
  nova.api.openstack[26783]: TRACE 
  nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 289, in decorated_function
  nova.api.openstack[26783]: TRACE     return function(self, context, *args, **kwargs)
  nova.api.openstack[26783]: TRACE 
  nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 331, in decorated_function
  nova.api.openstack[26783]: TRACE     kwargs['instance'], e, sys.exc_info())
  nova.api.openstack[26783]: TRACE 
  nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 82, in __exit__
  nova.api.openstack[26783]: TRACE     six.reraise(self.type_, self.value, self.tb)
  nova.api.openstack[26783]: TRACE 
  nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 319, in decorated_function
  nova.api.openstack[26783]: TRACE     return function(self, context, *args, **kwargs)
  nova.api.openstack[26783]: TRACE 
  nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4787, in attach_interface
  nova.api.openstack[26783]: TRACE     context, instance, port_id, network_id, requested_ip)
  nova.api.openstack[26783]: TRACE 
  nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 569, in allocate_port_for_instance
  nova.api.openstack[26783]: TRACE     requested_networks=requested_networks)
  nova.api.openstack[26783]: TRACE 
  nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 443, in allocate_for_instance
  nova.api.openstack[26783]: TRACE     self._delete_ports(neutron, instance, created_port_ids)
  nova.api.openstack[26783]: TRACE 
  nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 82, in __exit__
  nova.api.openstack[26783]: TRACE     six.reraise(self.type_, self.value, self.tb)
  nova.api.openstack[26783]: TRACE 
  nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 423, in allocate_for_instance
  nova.api.openstack[26783]: TRACE     security_group_ids, available_macs, dhcp_opts)
  nova.api.openstack[26783]: TRACE 
  nova.api.openstack[26783]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 226, in _create_port
  nova.api.openstack[26783]: TRACE     raise exception.NoMoreFixedIps()
  nova.api.openstack[26783]: TRACE 
  nova.api.openstack[26783]: TRACE NoMoreFixedIps: Zero fixed ips available.
  nova.api.openstack[26783]: TRACE 
  nova.api.openstack[26783]: TRACE 
  nova.api.openstack[26783]: INFO http://isokolov1.compute.dev.internal:8774/v2/78ea359977584bcc9feceef2553dbe57/servers/5f1e84cb-1766-45e1-899b-9de1e535309b/os-interface returned with HTTP 500 [req-57f4e821-a968-48cd-8358-f73fa16b4ff7 4aac5cb61b1741b2a32067619555ecc1 78ea359977584bcc9feceef2553dbe57]
  nova.api.openstack.wsgi[26783]: DEBUG Returning 500 to user: The server has either erred or is incapable of performing the requested operation. [req-57f4e821-a968-48cd-8358-f73fa16b4ff7 4aac5cb61b1741b2a32067619555ecc1 78ea359977584bcc9feceef2553dbe57] __call__ /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:1201
  nova.osapi_compute.wsgi.server[26783]: INFO 192.168.32.70 "POST /v2/78ea359977584bcc9feceef2553dbe57/servers/5f1e84cb-1766-45e1-899b-9de1e535309b/os-interface HTTP/1.1" status: 500 len: 335 time: 0.5942502 [req-57f4e821-a968-48cd-8358-f73fa16b4ff7 4aac5cb61b1741b2a32067619555ecc1 78ea359977584bcc9feceef2553dbe57]
  nova.api.openstack.wsgi[26782]: DEBUG Calling method '<bound method Controller.detail of <nova.api.openstack.compute.servers.Controller object at 0x7f098ebe6610>>' [req-f51d4aa9-0b6b-4b84-a247-04e650055589 44cc5bff123b4fe0b06f6ac90418756d b80de3539fd2406ea4c0d7c827c4fdd7] _process_stack /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:937
  nova.compute.api[26782]: DEBUG Searching by: {'deleted': False} [req-f51d4aa9-0b6b-4b84-a247-04e650055589 44cc5bff123b4fe0b06f6ac90418756d b80de3539fd2406ea4c0d7c827c4fdd7] get_all /usr/lib/python2.7/dist-packages/nova/compute/api.py:1920
  neutronclient.client[26782]: DEBUG 
  REQ: curl -i http://neutron-server.p.isokolov1.compute.serverscom.dev:29696/v2.0/ports.json?device_id=3ede6afc-7695-414c-bff0-4b807fe4644c&device_id=bf4abf26-b937-4f2d-8c29-6e69ad4be28c&device_id=375cd096-8022-4721-bcc9-5d65f1d21865&device_id=e8378206-ce54-45c3-976c-d3a38a8c619d&device_id=5f1e84cb-1766-45e1-899b-9de1e535309b -X GET -H "X-Auth-Token: bcc1f6eff61a434d9165794c082e7102" -H "User-Agent: python-neutronclient"
   [req-f51d4aa9-0b6b-4b84-a247-04e650055589 44cc5bff123b4fe0b06f6ac90418756d b80de3539fd2406ea4c0d7c827c4fdd7] http_log_req /usr/lib/python2.7/dist-packages/neutronclient/common/utils.py:138
  neutronclient.client[26782]: DEBUG RESP:200 CaseInsensitiveDict({'date': 'Mon, 23 Feb 2015 11:07:39 GMT', 'content-length': '2182', 'content-type': 'application/json; charset=UTF-8', 'x-openstack-request-id': 'req-809c50d1-d35c-4b0e-88c6-f9730d540216'}) {"ports": [{"status": "ACTIVE", "binding:host_id": "compute1", "allowed_address_pairs": [], "extra_dhcp_opts": [], "device_owner": "compute:nova", "binding:profile": {}, "fixed_ips": [{"subnet_id": "a4180ba0-6681-4e07-8cc8-0617cccfd12e", "ip_address": "191.168.1.10"}], "id": "11f8456d-41f9-46e7-b6c8-5f6becc59a07", "security_groups": ["0700a947-91aa-49a2-8cd6-c1e2abd8d8c4"], "device_id": "375cd096-8022-4721-bcc9-5d65f1d21865", "name": "", "admin_state_up": true, "network_id": "7ad70c6b-82f6-4723-be8e-d96047755f8e", "tenant_id": "b6749c7410594a01ae857f78d24884e3", "binding:vif_details": {"port_filter": true, "ovs_hybrid_plug": true}, "binding:vnic_type": "normal", "binding:vif_type": "ovs", "mac_address": "fa:16:3e:9a:09:e0"}, {"status": "ACTIVE", "binding:host_id": "compute1", "allowed_address_pairs": [], "extra_dhcp_opts": [], "device_owner": "compute:nova", "binding:profile": {}, "fixed_ips": [{"subnet_id": "85412a00-c7da-472a-8cfe-3f66f911263a", "ip_address": "192.168.1.1"}], "id": "3e66374d-54d2-4efc-be86-0c2359c90c7c", "security_groups": ["0700a947-91aa-49a2-8cd6-c1e2abd8d8c4"], "device_id": "3ede6afc-7695-414c-bff0-4b807fe4644c", "name": "", "admin_state_up": true, "network_id": "674ffba2-6393-4a77-85d2-cceab7960f28", "tenant_id": "b6749c7410594a01ae857f78d24884e3", "binding:vif_details": {"port_filter": true, "ovs_hybrid_plug": true}, "binding:vnic_type": "normal", "binding:vif_type": "ovs", "mac_address": "fa:16:3e:8c:ef:3a"}, {"status": "ACTIVE", "binding:host_id": "compute1", "allowed_address_pairs": [], "extra_dhcp_opts": [], "device_owner": "compute:nova", "binding:profile": {}, "fixed_ips": [{"subnet_id": "50ebc948-752c-48a4-9825-77a43deb1363", "ip_address": "193.168.1.10"}], "id": "a9deee70-6a99-4d4f-8e0c-9ba05a0d61f3", "security_groups": ["0700a947-91aa-49a2-8cd6-c1e2abd8d8c4"], "device_id": "bf4abf26-b937-4f2d-8c29-6e69ad4be28c", "name": "", "admin_state_up": true, "network_id": "f3f2802a-c2a1-4d8b-9f43-cf24d0dc8233", "tenant_id": "b6749c7410594a01ae857f78d24884e3", "binding:vif_details": {"port_filter": true, "ovs_hybrid_plug": true}, "binding:vnic_type": "normal", "binding:vif_type": "ovs", "mac_address": "fa:16:3e:a5:10:3e"}]}
   [req-f51d4aa9-0b6b-4b84-a247-04e650055589 44cc5bff123b4fe0b06f6ac90418756d b80de3539fd2406ea4c0d7c827c4fdd7] http_log_resp /usr/lib/python2.7/dist-packages/neutronclient/common/utils.py:147
  neutronclient.client[26782]: DEBUG 
  REQ: curl -i http://neutron-server.p.isokolov1.compute.serverscom.dev:29696/v2.0/security-groups.json?id=0700a947-91aa-49a2-8cd6-c1e2abd8d8c4 -X GET -H "X-Auth-Token: bcc1f6eff61a434d9165794c082e7102" -H "User-Agent: python-neutronclient"
   [req-f51d4aa9-0b6b-4b84-a247-04e650055589 44cc5bff123b4fe0b06f6ac90418756d b80de3539fd2406ea4c0d7c827c4fdd7] http_log_req /usr/lib/python2.7/dist-packages/neutronclient/common/utils.py:138
  neutronclient.client[26782]: DEBUG RESP:200 CaseInsensitiveDict({'date': 'Mon, 23 Feb 2015 11:07:39 GMT', 'content-length': '1535', 'content-type': 'application/json; charset=UTF-8', 'x-openstack-request-id': 'req-d048f970-115c-4ed1-a7d4-5c95d7a4642d'}) {"security_groups": [{"tenant_id": "b6749c7410594a01ae857f78d24884e3", "description": "default", "id": "0700a947-91aa-49a2-8cd6-c1e2abd8d8c4", "security_group_rules": [{"remote_group_id": "0700a947-91aa-49a2-8cd6-c1e2abd8d8c4", "direction": "ingress", "remote_ip_prefix": null, "protocol": null, "ethertype": "IPv4", "tenant_id": "b6749c7410594a01ae857f78d24884e3", "port_range_max": null, "port_range_min": null, "id": "259b62a6-7283-41db-8653-6578418aec00", "security_group_id": "0700a947-91aa-49a2-8cd6-c1e2abd8d8c4"}, {"remote_group_id": "0700a947-91aa-49a2-8cd6-c1e2abd8d8c4", "direction": "ingress", "remote_ip_prefix": null, "protocol": null, "ethertype": "IPv6", "tenant_id": "b6749c7410594a01ae857f78d24884e3", "port_range_max": null, "port_range_min": null, "id": "7b36cbe8-f616-4997-90b1-d0a684fff7f1", "security_group_id": "0700a947-91aa-49a2-8cd6-c1e2abd8d8c4"}, {"remote_group_id": null, "direction": "egress", "remote_ip_prefix": null, "protocol": null, "ethertype": "IPv4", "tenant_id": "b6749c7410594a01ae857f78d24884e3", "port_range_max": null, "port_range_min": null, "id": "9e9aa8e9-5bf8-427d-8c65-27022d39355f", "security_group_id": "0700a947-91aa-49a2-8cd6-c1e2abd8d8c4"}, {"remote_group_id": null, "direction": "egress", "remote_ip_prefix": null, "protocol": null, "ethertype": "IPv6", "tenant_id": "b6749c7410594a01ae857f78d24884e3", "port_range_max": null, "port_range_min": null, "id": "dbd6a976-0cf1-48d3-a437-6bf4a5dbb0ab", "security_group_id": "0700a947-91aa-49a2-8cd6-c1e2abd8d8c4"}], "name": "default"}]}
   [req-f51d4aa9-0b6b-4b84-a247-04e650055589 44cc5bff123b4fe0b06f6ac90418756d b80de3539fd2406ea4c0d7c827c4fdd7] http_log_resp /usr/lib/python2.7/dist-packages/neutronclient/common/utils.py:147
  nova.osapi_compute.wsgi.server[26782]: INFO 192.168.32.70 "GET /v2/b80de3539fd2406ea4c0d7c827c4fdd7/servers/detail?all_tenants=True HTTP/1.1" status: 200 len: 9633 time: 0.3986661 [req-f51d4aa9-0b6b-4b84-a247-04e650055589 44cc5bff123b4fe0b06f6ac90418756d b80de3539fd2406ea4c0d7c827c4fdd7]
  nova.api.openstack.wsgi[26781]: DEBUG Calling method '<bound method FloatingIPController.index of <nova.api.openstack.compute.contrib.floating_ips.FloatingIPController object at 0x7f098e02d890>>' [req-0383f71c-5767-482b-9ee4-53c0b4ff6806 44cc5bff123b4fe0b06f6ac90418756d b80de3539fd2406ea4c0d7c827c4fdd7] _process_stack /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:937
  neutronclient.client[26781]: DEBUG 
  REQ: curl -i http://neutron-server.p.isokolov1.compute.serverscom.dev:29696/v2.0/floatingips.json?tenant_id=b80de3539fd2406ea4c0d7c827c4fdd7 -X GET -H "X-Auth-Token: e775fc86f0f4418999241bfc779bc397" -H "User-Agent: python-neutronclient"
   [req-0383f71c-5767-482b-9ee4-53c0b4ff6806 44cc5bff123b4fe0b06f6ac90418756d b80de3539fd2406ea4c0d7c827c4fdd7] http_log_req /usr/lib/python2.7/dist-packages/neutronclient/common/utils.py:138
  neutronclient.client[26781]: DEBUG RESP:404 CaseInsensitiveDict({'date': 'Mon, 23 Feb 2015 11:07:40 GMT', 'content-length': '52', 'content-type': 'text/plain; charset=UTF-8', 'x-openstack-request-id': 'req-79af3c6f-c2d6-4899-af28-f31313a82dbe'}) 404 Not Found

  The resource could not be found.

     
   [req-0383f71c-5767-482b-9ee4-53c0b4ff6806 44cc5bff123b4fe0b06f6ac90418756d b80de3539fd2406ea4c0d7c827c4fdd7] http_log_resp /usr/lib/python2.7/dist-packages/neutronclient/common/utils.py:147
  neutronclient.v2_0.client[26781]: DEBUG Error message: 404 Not Found

  The resource could not be found.

      [req-0383f71c-5767-482b-9ee4-53c0b4ff6806 44cc5bff123b4fe0b06f6ac90418756d b80de3539fd2406ea4c0d7c827c4fdd7] _handle_fault_response /usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py:1208
  nova.api.openstack[26781]: ERROR Caught error: 404 Not Found

  The resource could not be found.

      [req-0383f71c-5767-482b-9ee4-53c0b4ff6806 44cc5bff123b4fe0b06f6ac90418756d b80de3539fd2406ea4c0d7c827c4fdd7]
  nova.api.openstack[26781]: TRACE Traceback (most recent call last):
  nova.api.openstack[26781]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 124, in __call__
  nova.api.openstack[26781]: TRACE     return req.get_response(self.application)
  nova.api.openstack[26781]: TRACE   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
  nova.api.openstack[26781]: TRACE     application, catch_exc_info=False)
  nova.api.openstack[26781]: TRACE   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
  nova.api.openstack[26781]: TRACE     app_iter = application(self.environ, start_response)
  nova.api.openstack[26781]: TRACE   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  nova.api.openstack[26781]: TRACE     return resp(environ, start_response)
  nova.api.openstack[26781]: TRACE   File "/usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py", line 661, in __call__
  nova.api.openstack[26781]: TRACE     return self._app(env, start_response)
  nova.api.openstack[26781]: TRACE   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  nova.api.openstack[26781]: TRACE     return resp(environ, start_response)
  nova.api.openstack[26781]: TRACE   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  nova.api.openstack[26781]: TRACE     return resp(environ, start_response)
  nova.api.openstack[26781]: TRACE   File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
  nova.api.openstack[26781]: TRACE     response = self.app(environ, start_response)
  nova.api.openstack[26781]: TRACE   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  nova.api.openstack[26781]: TRACE     return resp(environ, start_response)
  nova.api.openstack[26781]: TRACE   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
  nova.api.openstack[26781]: TRACE     resp = self.call_func(req, *args, **self.kwargs)
  nova.api.openstack[26781]: TRACE   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
  nova.api.openstack[26781]: TRACE     return self.func(req, *args, **kwargs)
  nova.api.openstack[26781]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 910, in __call__
  nova.api.openstack[26781]: TRACE     content_type, body, accept)
  nova.api.openstack[26781]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 975, in _process_stack
  nova.api.openstack[26781]: TRACE     action_result = self.dispatch(meth, request, action_args)
  nova.api.openstack[26781]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 1059, in dispatch
  nova.api.openstack[26781]: TRACE     return method(req=request, **action_args)
  nova.api.openstack[26781]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/floating_ips.py", line 135, in index
  nova.api.openstack[26781]: TRACE     floating_ips = self.network_api.get_floating_ips_by_project(context)
  nova.api.openstack[26781]: TRACE   File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 1096, in get_floating_ips_by_project
  nova.api.openstack[26781]: TRACE     fips = client.list_floatingips(tenant_id=project_id)['floatingips']
  nova.api.openstack[26781]: TRACE   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 99, in with_params
  nova.api.openstack[26781]: TRACE     ret = self.function(instance, *args, **kwargs)
  nova.api.openstack[26781]: TRACE   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 444, in list_floatingips
  nova.api.openstack[26781]: TRACE     **_params)
  nova.api.openstack[26781]: TRACE   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1335, in list
  nova.api.openstack[26781]: TRACE     for r in self._pagination(collection, path, **params):
  nova.api.openstack[26781]: TRACE   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1348, in _pagination
  nova.api.openstack[26781]: TRACE     res = self.get(path, params=params)
  nova.api.openstack[26781]: TRACE   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1321, in get
  nova.api.openstack[26781]: TRACE     headers=headers, params=params)
  nova.api.openstack[26781]: TRACE   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1298, in retry_request
  nova.api.openstack[26781]: TRACE     headers=headers, params=params)
  nova.api.openstack[26781]: TRACE   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1252, in do_request
  nova.api.openstack[26781]: TRACE     self._handle_fault_response(status_code, replybody)
  nova.api.openstack[26781]: TRACE   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1217, in _handle_fault_response
  nova.api.openstack[26781]: TRACE     exception_handler_v20(status_code, des_error_body)
  nova.api.openstack[26781]: TRACE   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 80, in exception_handler_v20
  nova.api.openstack[26781]: TRACE     message=message)
  nova.api.openstack[26781]: TRACE NeutronClientException: 404 Not Found
  nova.api.openstack[26781]: TRACE 
  nova.api.openstack[26781]: TRACE The resource could not be found.
  nova.api.openstack[26781]: TRACE 
  nova.api.openstack[26781]: TRACE    
  nova.api.openstack[26781]: TRACE 
  nova.api.openstack[26781]: INFO http://isokolov1.compute.dev.internal:8774/v2/b80de3539fd2406ea4c0d7c827c4fdd7/os-floating-ips returned with HTTP 500 [req-0383f71c-5767-482b-9ee4-53c0b4ff6806 44cc5bff123b4fe0b06f6ac90418756d b80de3539fd2406ea4c0d7c827c4fdd7]
  nova.api.openstack.wsgi[26781]: DEBUG Returning 500 to user: The server has either erred or is incapable of performing the requested operation. [req-0383f71c-5767-482b-9ee4-53c0b4ff6806 44cc5bff123b4fe0b06f6ac90418756d b80de3539fd2406ea4c0d7c827c4fdd7] __call__ /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:1201

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1424594/+subscriptions
Follow ups

[Bug 1424594] Re: 500 error and 2 traces if no free fixed IP left in the neutron network
From: Thierry Carrez, 2015-03-20
[Bug 1424594] [NEW] 500 error and 2 traces if no free fixed IP left in the neutron network
From: George Shuklin, 2015-02-23
References

[Bug 1424594] [NEW] 500 error and 2 traces if no free fixed IP left in the neutron network
From: George Shuklin, 2015-02-23