openstack team mailing list archive

Thread
Date

Multi-Zone finally working on ESSEX but cant "nova list" (KeyError: 'uuid') + doubts

To: openstack@xxxxxxxxxxxxxxxxxxx
From: Alejandro Comisario <alejandro.comisario@xxxxxxxxxxxxxxxx>
Date: Wed, 25 Jan 2012 17:58:30 -0300
User-agent: Mozilla/5.0 (X11; Linux i686; rv:9.0) Gecko/20111222 Thunderbird/9.0.1

Hey guys, we finally got MultiZone working on Essex with KVM ashypervisor (trunk version 2012.1-dev (2012.1-LOCALBRANCH:LOCALREVISION))we have one Parent zone and one child zone (named "zoneE") allintegrated with keystone ( nova, glance ) and we are able to spawninstances across both zones !After making this work after a couple of days of work, configuration anddebbuging we have one doubt, and one ERROR.

So, lets hope you can help us on both of them.

THE DOUBT:

This is deffinitely because we dont clearly understand the whole conceptbut, why is that "nova.scheduler.least_cost.weighted_sum" functionchooses as prefered host the LESS RAM memory host ? this seems to choosejust one host till is fully depleted, and then, chooses the other one !that doesnt sound like a good way of balancing instances across thezones by default, so, since we didnt know how to fix that behavior, justadded a "reverse=True" in the sorted functions, so problem "fixed".The thing is, if there is (maybe trough nova.conf) a way of controllingthis ( the sort order ), can any one shed some light on how to do it ?


THE ERROR / BUG:

At the moment that you spawn successfully an instance from the parentzone, to the child zone (the instance is successfully spawned andrunning on the child zone) the "nova list" (or curl to server/detail) tothe parent zone just BROKE.It apperars that when the parent zone calls the "server/detail" of thechild zone, the json returned by the child, has the "uuid" field printedinto the "id" field, so you can see on nova-api.log, having a keyerror"uuid" that breaks the call, you can check the pasteBin link below, oryou can keep reading and watch the nova-api.log stack from here.


Waiting for some feedback !
Cheers!
Alejandro.

PASTEBIN LINK:
--------------
http://pastebin.com/VGt8A3HX

ENTIRE NOVA-API STACK TRACE:
----------------------------

2012-01-25 14:42:15,793 DEBUG nova [-] HTTP PERF: 0.10457 seconds to GET172.16.146.139:35357 /v2.0/tokens/8de5e217-9615-443c-8b7e-09cabf4b4af8)from (pid=7488) getresponse/usr/local/lib/python2.6/dist-packages/keystone-1.0-py2.6.egg/keystone/common/bufferedhttp.py:992012-01-25 14:42:15,894 DEBUG nova [-] HTTP PERF: 0.10093 seconds to GET172.16.146.139:35357 /v2.0/tokens/8de5e217-9615-443c-8b7e-09cabf4b4af8)from (pid=7488) getresponse/usr/local/lib/python2.6/dist-packages/keystone-1.0-py2.6.egg/keystone/common/bufferedhttp.py:992012-01-25 14:42:15,897 DEBUG routes.middleware[2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin9574193127d1439fb09d1c0ef3eb26d5] Matched GET/9574193127d1439fb09d1c0ef3eb26d5/servers/detail from (pid=7488)__call__ /usr/lib/pymodules/python2.6/routes/middleware.py:1002012-01-25 14:42:15,897 DEBUG routes.middleware[2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin9574193127d1439fb09d1c0ef3eb26d5] Route path:'/:(project_id)/servers/:(id)', defaults: {'action': u'process','controller': <nova.api.openstack.v2.extensions.RequestExtensionResourceobject at 0x261b850>} from (pid=7488) __call__/usr/lib/pymodules/python2.6/routes/middleware.py:1022012-01-25 14:42:15,898 DEBUG routes.middleware[2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin9574193127d1439fb09d1c0ef3eb26d5] Match dict: {'action': u'process','controller': <nova.api.openstack.v2.extensions.RequestExtensionResourceobject at 0x261b850>, 'project_id': u'9574193127d1439fb09d1c0ef3eb26d5','id': u'detail'} from (pid=7488) __call__/usr/lib/pymodules/python2.6/routes/middleware.py:1032012-01-25 14:42:15,899 INFO nova.api.openstack.wsgi[2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin9574193127d1439fb09d1c0ef3eb26d5] GEThttp://172.16.162.5:8774/v1.1/9574193127d1439fb09d1c0ef3eb26d5/servers/detail2012-01-25 14:42:15,899 DEBUG nova.api.openstack.wsgi[2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin9574193127d1439fb09d1c0ef3eb26d5] Unrecognized Content-Type provided inrequest from (pid=7488) deserialize_body/usr/lib/python2.6/dist-packages/nova/api/openstack/wsgi.py:2572012-01-25 14:42:15,900 DEBUG routes.middleware[2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin9574193127d1439fb09d1c0ef3eb26d5] Matched GET/9574193127d1439fb09d1c0ef3eb26d5/servers/detail from (pid=7488)__call__ /usr/lib/pymodules/python2.6/routes/middleware.py:1002012-01-25 14:42:15,901 DEBUG routes.middleware[2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin9574193127d1439fb09d1c0ef3eb26d5] Route path:'/{project_id}/servers/detail', defaults: {'action': u'detail','controller': <nova.api.openstack.wsgi.Resource object at 0x26d5a50>}from (pid=7488) __call__/usr/lib/pymodules/python2.6/routes/middleware.py:1022012-01-25 14:42:15,902 DEBUG routes.middleware[2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin9574193127d1439fb09d1c0ef3eb26d5] Match dict: {'action': u'detail','controller': <nova.api.openstack.wsgi.Resource object at 0x26d5a50>,'project_id': u'9574193127d1439fb09d1c0ef3eb26d5'} from (pid=7488)__call__ /usr/lib/pymodules/python2.6/routes/middleware.py:1032012-01-25 14:42:15,902 INFO nova.api.openstack.wsgi[2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin9574193127d1439fb09d1c0ef3eb26d5] GEThttp://172.16.162.5:8774/v1.1/9574193127d1439fb09d1c0ef3eb26d5/servers/detail2012-01-25 14:42:15,903 DEBUG nova.api.openstack.wsgi[2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin9574193127d1439fb09d1c0ef3eb26d5] Unrecognized Content-Type provided inrequest from (pid=7488) deserialize_body/usr/lib/python2.6/dist-packages/nova/api/openstack/wsgi.py:2572012-01-25 14:42:15,903 DEBUG nova.compute.api[2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin9574193127d1439fb09d1c0ef3eb26d5] Searching by: {'deleted': False,'local_zone_only': False} from (pid=7488) get_all/usr/lib/python2.6/dist-packages/nova/compute/api.py:9372012-01-25 14:42:16,116 DEBUG novaclient.client[2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin9574193127d1439fb09d1c0ef3eb26d5] REQ: curl -ihttp://172.16.146.139:5000/v2.0/tokens -X POST -H "Content-Type:application/json" -H "User-Agent: python-novaclient"from (pid=7488) http_log/usr/lib/python2.6/dist-packages/novaclient/client.py:762012-01-25 14:42:16,116 DEBUG novaclient.client[2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin9574193127d1439fb09d1c0ef3eb26d5] REQ BODY: {"auth":{"passwordCredentials": {"username": "esmadmin", "password": "123456"}}}from (pid=7488) http_log/usr/lib/python2.6/dist-packages/novaclient/client.py:782012-01-25 14:42:16,117 DEBUG novaclient.client[2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin9574193127d1439fb09d1c0ef3eb26d5] RESP:{'date': 'Wed, 25 Jan 201219:42:16 GMT', 'vary': 'X-Auth-Token', 'content-length': '1583','status': '200', 'content-type': 'application/json; charset=UTF-8'}{"access": {"token": {"expires": "2012-01-26T15:42:15.970710","tenants": [{"id": "df58327d2b6c4e86b1df9b4671c5b54b", "name":"esmadmin"}], "id": "d9164294-0e8f-4f01-94c6-eb4b460c95ee", "tenant":{"id": "df58327d2b6c4e86b1df9b4671c5b54b", "name": "esmadmin"}},"serviceCatalog": [{"endpoints": [{"adminURL":"http://172.16.163.5:8774/v1.1/df58327d2b6c4e86b1df9b4671c5b54b";,"region": "zoneE", "id": "5", "internalURL":"http://172.16.163.5:8774/v1.1/df58327d2b6c4e86b1df9b4671c5b54b";,"publicURL":"http://172.16.163.5:8774/v1.1/df58327d2b6c4e86b1df9b4671c5b54b"}],"type": "compute", "name": "nova"}, {"endpoints": [{"adminURL":"http://172.16.147.123/v1.1/df58327d2b6c4e86b1df9b4671c5b54b";, "region":"RegionOne", "id": "2", "internalURL":"http://172.16.147.123:9292/v1.1/df58327d2b6c4e86b1df9b4671c5b54b";,"publicURL":"http://172.16.147.123/v1.1/df58327d2b6c4e86b1df9b4671c5b54b"}], "type":"image", "name": "glance"}, {"endpoints": [{"adminURL":"http://172.16.146.139:35357/v2.0";, "region": "RegionOne", "id": "3","internalURL": "http://172.16.146.139:5000/v2.0";, "publicURL":"http://172.16.146.139:5000/v2.0"}], "type": "identity", "name":"keystone"}, {"endpoints": [{"adminURL": "http://172.16.0.88:8080/";,"region": "RegionOne", "id": "4", "internalURL":"http://172.16.0.88:8080/v1/AUTH_df58327d2b6c4e86b1df9b4671c5b54b";,"publicURL":"http://172.16.0.88:8080/v1/AUTH_df58327d2b6c4e86b1df9b4671c5b54b"}],"type": "object-store", "name": "swift"}], "user": {"id":"4a4774f4374b4a928df9319009302a34", "roles": [{"name": "KeystoneAdmin","id": "3"}], "name": "esmadmin"}}}from (pid=7488) http_log/usr/lib/python2.6/dist-packages/novaclient/client.py:792012-01-25 14:42:16,119 DEBUG novaclient.client[2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin9574193127d1439fb09d1c0ef3eb26d5] Using Endpoint URL:http://172.16.146.139:35357/v2.0/tokens/8de5e217-9615-443c-8b7e-09cabf4b4af8/endpointsfrom (pid=7488) _fetch_endpoints_from_auth/usr/lib/python2.6/dist-packages/novaclient/client.py:1852012-01-25 14:42:16,238 DEBUG novaclient.client[2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin9574193127d1439fb09d1c0ef3eb26d5] REQ: curl -ihttp://172.16.146.139:35357/v2.0/tokens/8de5e217-9615-443c-8b7e-09cabf4b4af8/endpoints-X GET -H "X-Auth_Token: d9164294-0e8f-4f01-94c6-eb4b460c95ee" -H"User-Agent: python-novaclient"from (pid=7488) http_log/usr/lib/python2.6/dist-packages/novaclient/client.py:762012-01-25 14:42:16,239 DEBUG novaclient.client[2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin9574193127d1439fb09d1c0ef3eb26d5] RESP:{'status': '200','content-length': '870', 'content-location':'http://172.16.146.139:35357/v2.0/tokens/8de5e217-9615-443c-8b7e-09cabf4b4af8/endpoints','vary': 'X-Auth-Token', 'date': 'Wed, 25 Jan 2012 19:42:16 GMT','content-type': 'application/json; charset=UTF-8'} {"endpoints_links":[{"href":"http://172.16.146.139:35357/tokens/8de5e217-9615-443c-8b7e-09cabf4b4af8/endpoints?marker=3&limit=10";,"rel": "next"}], "endpoints": [{"internalURL":"http://172.16.162.5:8774/v1.1/9574193127d1439fb09d1c0ef3eb26d5";,"name": "nova", "adminURL":"http://172.16.162.5:8774/v1.1/9574193127d1439fb09d1c0ef3eb26d5";,"region": "RegionOne", "tenantId": "9574193127d1439fb09d1c0ef3eb26d5","type": "compute", "id": 2, "publicURL":"http://172.16.162.5:8774/v1.1/9574193127d1439fb09d1c0ef3eb26d5"},{"internalURL":"http://172.16.163.5:8774/v1.1/9574193127d1439fb09d1c0ef3eb26d5";,"name": "nova", "adminURL":"http://172.16.163.5:8774/v1.1/9574193127d1439fb09d1c0ef3eb26d5";,"region": "zoneE", "tenantId": "9574193127d1439fb09d1c0ef3eb26d5","type": "compute", "id": 3, "publicURL":"http://172.16.163.5:8774/v1.1/9574193127d1439fb09d1c0ef3eb26d5"}]}from (pid=7488) http_log/usr/lib/python2.6/dist-packages/novaclient/client.py:792012-01-25 14:42:16,737 DEBUG novaclient.client [-] REQ: curl -ihttp://172.16.163.5:8774/v1.1/9574193127d1439fb09d1c0ef3eb26d5/servers/detail-X GET -H "User-Agent: python-novaclient" -H "X-Auth-Token:8de5e217-9615-443c-8b7e-09cabf4b4af8"from (pid=7488) http_log/usr/lib/python2.6/dist-packages/novaclient/client.py:762012-01-25 14:42:16,738 DEBUG novaclient.client [-] RESP:{'date': 'Wed,25 Jan 2012 19:42:16 GMT', 'status': '200', 'content-length': '2397','content-type': 'application/json', 'content-location':u'http://172.16.163.5:8774/v1.1/9574193127d1439fb09d1c0ef3eb26d5/servers/detail'}{"servers": [{"OS-EXT-STS:task_state": null, "addresses": {}, "links":[{"href":"http://172.16.163.5:8774/v1.1/9574193127d1439fb09d1c0ef3eb26d5/servers/d40dc517-3eba-4eaa-bf2a-b15fac8ee2c4";,"rel": "self"}, {"href":"http://172.16.163.5:8774/9574193127d1439fb09d1c0ef3eb26d5/servers/d40dc517-3eba-4eaa-bf2a-b15fac8ee2c4";,"rel": "bookmark"}], "image": {"id":"b05ba8d5-9f71-40df-851f-709771da62a9", "links": [{"href":"http://172.16.163.5:8774/9574193127d1439fb09d1c0ef3eb26d5/images/b05ba8d5-9f71-40df-851f-709771da62a9";,"rel": "bookmark"}]}, "OS-EXT-STS:vm_state": "active", "flavor": {"id":"1", "links": [{"href":"http://172.16.163.5:8774/9574193127d1439fb09d1c0ef3eb26d5/flavors/1";,"rel": "bookmark"}]}, "id": "d40dc517-3eba-4eaa-bf2a-b15fac8ee2c4","user_id": "dsmadmin", "RAX-DCF:diskConfig": "MANUAL", "accessIPv4": "","accessIPv6": "", "progress": null, "OS-EXT-STS:power_state": 1,"config_drive": "", "status": "ACTIVE", "updated":"2012-01-25T19:41:15Z", "hostId":"64fe9962f6dc6c376f4e8f9eb6c871aaf0da3771d9a415f0bf60a770", "key_name":"", "name": "vmtest1", "created": "2012-01-25T19:40:57Z", "tenant_id":"9574193127d1439fb09d1c0ef3eb26d5", "metadata": {"My Server Name":"vmtest1"}}, {"OS-EXT-STS:task_state": null, "addresses": {}, "links":[{"href":"http://172.16.163.5:8774/v1.1/9574193127d1439fb09d1c0ef3eb26d5/servers/8f00967a-bb2a-4f24-bf5d-48b1f3c77388";,"rel": "self"}, {"href":"http://172.16.163.5:8774/9574193127d1439fb09d1c0ef3eb26d5/servers/8f00967a-bb2a-4f24-bf5d-48b1f3c77388";,"rel": "bookmark"}], "image": {"id":"b05ba8d5-9f71-40df-851f-709771da62a9", "links": [{"href":"http://172.16.163.5:8774/9574193127d1439fb09d1c0ef3eb26d5/images/b05ba8d5-9f71-40df-851f-709771da62a9";,"rel": "bookmark"}]}, "OS-EXT-STS:vm_state": "active", "flavor": {"id":"1", "links": [{"href":"http://172.16.163.5:8774/9574193127d1439fb09d1c0ef3eb26d5/flavors/1";,"rel": "bookmark"}]}, "id": "8f00967a-bb2a-4f24-bf5d-48b1f3c77388","user_id": "dsmadmin", "RAX-DCF:diskConfig": "MANUAL", "accessIPv4": "","accessIPv6": "", "progress": null, "OS-EXT-STS:power_state": 1,"config_drive": "", "status": "ACTIVE", "updated":"2012-01-25T19:38:31Z", "hostId":"028036137ae671554f81f2262ee84138a1f375d32b91550a34fdaca1", "key_name":"", "name": "vmtest1", "created": "2012-01-25T19:38:22Z", "tenant_id":"9574193127d1439fb09d1c0ef3eb26d5", "metadata": {"My Server Name":"vmtest1"}}]}from (pid=7488) http_log/usr/lib/python2.6/dist-packages/novaclient/client.py:792012-01-25 14:42:16,741 ERROR nova.api.openstack.v2[2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin9574193127d1439fb09d1c0ef3eb26d5] Caught error: 'uuid'

(nova.api.openstack.v2): TRACE: Traceback (most recent call last):

(nova.api.openstack.v2): TRACE: File"/usr/lib/python2.6/dist-packages/nova/api/openstack/v2/__init__.py",line 62, in __call__(nova.api.openstack.v2): TRACE: returnreq.get_response(self.application)(nova.api.openstack.v2): TRACE: File"/usr/lib/pymodules/python2.6/webob/request.py", line 1053, in get_response

(nova.api.openstack.v2): TRACE:     application, catch_exc_info=False)

(nova.api.openstack.v2): TRACE: File"/usr/lib/pymodules/python2.6/webob/request.py", line 1022, incall_application(nova.api.openstack.v2): TRACE: app_iter = application(self.environ,start_response)(nova.api.openstack.v2): TRACE: File"/usr/local/lib/python2.6/dist-packages/keystone-1.0-py2.6.egg/keystone/middleware/auth_token.py",line 186, in __call__(nova.api.openstack.v2): TRACE: return self._forward_request(env,start_response, proxy_headers)(nova.api.openstack.v2): TRACE: File"/usr/local/lib/python2.6/dist-packages/keystone-1.0-py2.6.egg/keystone/middleware/auth_token.py",line 312, in _forward_request

(nova.api.openstack.v2): TRACE:     return self.app(env, start_response)

(nova.api.openstack.v2): TRACE: File"/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__

(nova.api.openstack.v2): TRACE:     return resp(environ, start_response)

(nova.api.openstack.v2): TRACE: File"/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__

(nova.api.openstack.v2): TRACE:     return resp(environ, start_response)

(nova.api.openstack.v2): TRACE: File"/usr/lib/pymodules/python2.6/webob/dec.py", line 147, in __call__(nova.api.openstack.v2): TRACE: resp = self.call_func(req, *args,**self.kwargs)(nova.api.openstack.v2): TRACE: File"/usr/lib/pymodules/python2.6/webob/dec.py", line 208, in call_func

(nova.api.openstack.v2): TRACE:     return self.func(req, *args, **kwargs)

(nova.api.openstack.v2): TRACE: File"/usr/lib/python2.6/dist-packages/nova/api/openstack/wsgi.py", line 499,in __call__(nova.api.openstack.v2): TRACE: response =req.get_response(self.application)(nova.api.openstack.v2): TRACE: File"/usr/lib/pymodules/python2.6/webob/request.py", line 1053, in get_response

(nova.api.openstack.v2): TRACE:     application, catch_exc_info=False)

(nova.api.openstack.v2): TRACE:     return resp(environ, start_response)

(nova.api.openstack.v2): TRACE: File"/usr/lib/pymodules/python2.6/routes/middleware.py", line 131, in __call__(nova.api.openstack.v2): TRACE: response = self.app(environ,start_response)(nova.api.openstack.v2): TRACE: File"/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__

(nova.api.openstack.v2): TRACE:     return resp(environ, start_response)

(nova.api.openstack.v2): TRACE:     return self.func(req, *args, **kwargs)

(nova.api.openstack.v2): TRACE:     res = req.get_response(self.application)

(nova.api.openstack.v2): TRACE: File"/usr/lib/pymodules/python2.6/webob/request.py", line 1053, in get_response

(nova.api.openstack.v2): TRACE:     application, catch_exc_info=False)

(nova.api.openstack.v2): TRACE:     return resp(environ, start_response)

(nova.api.openstack.v2): TRACE:     return resp(environ, start_response)

(nova.api.openstack.v2): TRACE:     return self.func(req, *args, **kwargs)

(nova.api.openstack.v2): TRACE: File"/usr/lib/python2.6/dist-packages/nova/api/openstack/wsgi.py", line 567,in __call__(nova.api.openstack.v2): TRACE: action_result =self.dispatch(request, action, args)(nova.api.openstack.v2): TRACE: File"/usr/lib/python2.6/dist-packages/nova/api/openstack/wsgi.py", line 599,in dispatch(nova.api.openstack.v2): TRACE: returncontroller_method(req=request, **action_args)(nova.api.openstack.v2): TRACE: File"/usr/lib/python2.6/dist-packages/nova/api/openstack/v2/servers.py",line 75, in detail(nova.api.openstack.v2): TRACE: servers = self._get_servers(req,is_detail=True)(nova.api.openstack.v2): TRACE: File"/usr/lib/python2.6/dist-packages/nova/api/openstack/v2/servers.py",line 150, in _get_servers(nova.api.openstack.v2): TRACE: self._add_instance_faults(context,limited_list)(nova.api.openstack.v2): TRACE: File"/usr/lib/python2.6/dist-packages/nova/api/openstack/v2/servers.py",line 89, in _add_instance_faults(nova.api.openstack.v2): TRACE: faults =self.compute_api.get_instance_faults(ctxt, instances)(nova.api.openstack.v2): TRACE: File"/usr/lib/python2.6/dist-packages/nova/compute/api.py", line 1607, inget_instance_faults(nova.api.openstack.v2): TRACE: uuids = [instance['uuid'] forinstance in instances]

(nova.api.openstack.v2): TRACE: KeyError: 'uuid'
(nova.api.openstack.v2): TRACE: