openstack team mailing list archive

Thread
Date

essex rc1: rabbit time out when runing nova x509-get-root-cert

To: openstack <openstack@xxxxxxxxxxxxxxxxxxx>
From: Pierre Amadio <pierre.amadio@xxxxxxxxxxxxx>
Date: Mon, 26 Mar 2012 14:38:20 +0200
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:11.0) Gecko/20120310 Thunderbird/11.0

Hi there !

I am trying to have a 4 node essex-rc1 setup running
nova/kesytone/glance (no swift, no horizon).

Things have been installed from the milestone-proposed branch of each
project.

I reach the stage where some nova command works (such as nova
flavor-list), but i cannot get a root certificate with

nova x509-get-root-cert

It looks to me nova-api send the request to rabbit, but that nobody
reply to it.

You will find more details (environment variables, nova-api logs, nova
--debug output and mostly empty rabbit log) in the attached pb-cert.txt
file.

Any idea what could be wrong ?

192.168.122.101: manager node (nova-api, nova-objectstore,nova-scheduler,nova-volume)
192.168.122.102: messaging (rabbit, keystone, glance, mysql)
192.168.122.103: nova-compute nova-network
192.168.122.104: nova-compute


nova command seems to work:

ubuntu@manager-node:~/creds$ cat openrc 
# OS API settings
export NOVA_USERNAME=ubuntu
export NOVA_PROJECT_ID=ubuntu
export NOVA_PASSWORD=openstack
export NOVA_API_KEY=${NOVA_PASSWORD}
#where keystone runs
export NOVA_URL=http://192.168.122.102:35357/v2.0/
export NOVA_VERSION=1.1
export NOVA_REGION_NAME=RegionOne
export OS_AUTH_USER=${NOVA_USERNAME}
export OS_AUTH_KEY=${NOVA_PASSWORD}
export OS_AUTH_TENANT=${NOVA_PROJECT_ID}
export OS_AUTH_URL=${NOVA_URL}
export OS_AUTH_STRATEGY=keystone

# EC2 API settings
# Set the ec2 url so euca2ools works
export EC2_URL=${EC2_URL:-http://192.168.122.101:8773/services/Cloud}

# Access key is set in the initial keystone data to be the same as username
export EC2_ACCESS_KEY=3afad56c1e47481991e70b7094650d1a

# Secret key is set in the initial keystone data to the admin password
export EC2_SECRET_KEY=59cf599d36264764a7bd288ed87136bc

#export EC2_CERT=/home/ubuntu/creds/cacert.pem
#export EC2_PRIVATE_KEY=/home/ubuntu/creds/cert.pem


ubuntu@manager-node:~/creds$ . openrc
ubuntu@manager-node:~/creds$ nova flavor-list
+----+-----------+-----------+------+-----------+------+-------+-------------+
| ID |    Name   | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor |
+----+-----------+-----------+------+-----------+------+-------+-------------+
| 1  | m1.tiny   | 512       | 0    | 0         |      | 1     | 1.0         |
| 2  | m1.small  | 2048      | 10   | 20        |      | 1     | 1.0         |
| 3  | m1.medium | 4096      | 10   | 40        |      | 2     | 1.0         |
| 4  | m1.large  | 8192      | 10   | 80        |      | 4     | 1.0         |
| 5  | m1.xlarge | 16384     | 10   | 160       |      | 8     | 1.0         |
+----+-----------+-----------+------+-----------+------+-------+-------------+

But i experience a time out when trying to launch the following:



ubuntu@manager-node:~/creds$ nova --debug x509-get-root-cert
connect: (192.168.122.102, 35357)
send: 'POST /v2.0/tokens HTTP/1.1\r\nHost: 192.168.122.102:35357\r\nContent-Length: 106\r\ncontent-type: application/json\r\naccept-encoding: gzip, deflate\r\naccept: application/json\r\nuser-agent: python-novaclient\r\n\r\n{"auth": {"tenantName": "ubuntu", "passwordCredentials": {"username": "ubuntu", "password": "openstack"}}}'
reply: 'HTTP/1.1 200 OK\r\n'
header: Content-Type: application/json
header: Vary: X-Auth-Token
header: Content-Length: 1645
header: Date: Mon, 26 Mar 2012 11:44:36 GMT
connect: (192.168.122.101, 8774)
send: u'GET /v1.1/a07fd73258684caaa86edcaabd7a16b0/os-certificates/root HTTP/1.1\r\nHost: 192.168.122.101:8774\r\nx-auth-project-id: ubuntu\r\nx-auth-token: b59aedb8cca04336a47dec804a382657\r\naccept-encoding: gzip, deflate\r\naccept: application/json\r\nuser-agent: python-novaclient\r\n\r\n'
reply: 'HTTP/1.1 500 Internal Server Error\r\n'
header: Content-Length: 128
header: Content-Type: application/json; charset=UTF-8
header: X-Compute-Request-Id: req-161614ff-cf70-4aba-a8f2-9116924eec7b
header: Date: Mon, 26 Mar 2012 11:45:36 GMT
DEBUG (shell:416) The server has either erred or is incapable of performing the requested operation. (HTTP 500)
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/python_novaclient-2012.1-py2.7.egg/novaclient/shell.py", line 413, in main
    OpenStackComputeShell().main(sys.argv[1:])
  File "/usr/local/lib/python2.7/dist-packages/python_novaclient-2012.1-py2.7.egg/novaclient/shell.py", line 364, in main
    args.func(self.cs, args)
  File "/usr/local/lib/python2.7/dist-packages/python_novaclient-2012.1-py2.7.egg/novaclient/v1_1/shell.py", line 1458, in do_x509_get_root_cert
    cacert = cs.certs.get()
  File "/usr/local/lib/python2.7/dist-packages/python_novaclient-2012.1-py2.7.egg/novaclient/v1_1/certs.py", line 48, in get
    return self._get("/os-certificates/root", 'certificate')
  File "/usr/local/lib/python2.7/dist-packages/python_novaclient-2012.1-py2.7.egg/novaclient/base.py", line 149, in _get
    resp, body = self.api.client.get(url)
  File "/usr/local/lib/python2.7/dist-packages/python_novaclient-2012.1-py2.7.egg/novaclient/client.py", line 136, in get
    return self._cs_request(url, 'GET', **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/python_novaclient-2012.1-py2.7.egg/novaclient/client.py", line 124, in _cs_request
    **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/python_novaclient-2012.1-py2.7.egg/novaclient/client.py", line 107, in request
    raise exceptions.from_response(resp, body)
ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500)
ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500)


######################################################################################################################################"
Rabbit log

=INFO REPORT==== 26-Mar-2012::13:44:36 ===
starting TCP connection <0.479.0> from 192.168.122.101:41802

######################################################################################################################################
nova-api log:
2012-03-26 13:44:36 INFO nova.api.openstack.wsgi [req-161614ff-cf70-4aba-a8f2-9116924eec7b 8e5902533835427b8f812ec60c909546 a07fd73258684caaa86edcaabd7a16b0] GET http://192.168.122.101:8774/v1.1/a07fd73258684caaa86edcaabd7a16b0/os-certificates/root
2012-03-26 13:44:36 DEBUG nova.api.openstack.wsgi [req-161614ff-cf70-4aba-a8f2-9116924eec7b 8e5902533835427b8f812ec60c909546 a07fd73258684caaa86edcaabd7a16b0] Unrecognized Content-Type provided in request from (pid=14918) get_body /usr/local/lib/python2.7/dist-packages/nova-2012.1-py2.7.egg/nova/api/openstack/wsgi.py:697
2012-03-26 13:44:36 DEBUG nova.rpc.amqp [req-161614ff-cf70-4aba-a8f2-9116924eec7b 8e5902533835427b8f812ec60c909546 a07fd73258684caaa86edcaabd7a16b0] Making asynchronous call on cert ... from (pid=14918) multicall /usr/local/lib/python2.7/dist-packages/nova-2012.1-py2.7.egg/nova/rpc/amqp.py:321
2012-03-26 13:44:36 DEBUG nova.rpc.amqp [req-161614ff-cf70-4aba-a8f2-9116924eec7b 8e5902533835427b8f812ec60c909546 a07fd73258684caaa86edcaabd7a16b0] MSG_ID is dce3eaea73bb4c6fb0e592c075b07562 from (pid=14918) multicall /usr/local/lib/python2.7/dist-packages/nova-2012.1-py2.7.egg/nova/rpc/amqp.py:324
2012-03-26 13:44:36 DEBUG nova.rpc.amqp [req-161614ff-cf70-4aba-a8f2-9116924eec7b 8e5902533835427b8f812ec60c909546 a07fd73258684caaa86edcaabd7a16b0] Pool creating new connection from (pid=14918) create /usr/local/lib/python2.7/dist-packages/nova-2012.1-py2.7.egg/nova/rpc/amqp.py:59
2012-03-26 13:44:36 INFO nova.rpc.common [req-161614ff-cf70-4aba-a8f2-9116924eec7b 8e5902533835427b8f812ec60c909546 a07fd73258684caaa86edcaabd7a16b0] Connected to AMQP server on 192.168.122.102:5672
2012-03-26 13:45:36 ERROR nova.rpc.common [req-161614ff-cf70-4aba-a8f2-9116924eec7b 8e5902533835427b8f812ec60c909546 a07fd73258684caaa86edcaabd7a16b0] Timed out waiting for RPC response: timed out
(nova.rpc.common): TRACE: Traceback (most recent call last):
(nova.rpc.common): TRACE:   File "/usr/local/lib/python2.7/dist-packages/nova-2012.1-py2.7.egg/nova/rpc/impl_kombu.py", line 490, in ensure
(nova.rpc.common): TRACE:     return method(*args, **kwargs)
(nova.rpc.common): TRACE:   File "/usr/local/lib/python2.7/dist-packages/nova-2012.1-py2.7.egg/nova/rpc/impl_kombu.py", line 567, in _consume
(nova.rpc.common): TRACE:     return self.connection.drain_events(timeout=timeout)
(nova.rpc.common): TRACE:   File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 110, in drain_events
(nova.rpc.common): TRACE:     return self.transport.drain_events(self.connection, **kwargs)
(nova.rpc.common): TRACE:   File "/usr/local/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 200, in drain_events
(nova.rpc.common): TRACE:     return connection.drain_events(**kwargs)
(nova.rpc.common): TRACE:   File "/usr/local/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 50, in drain_events
(nova.rpc.common): TRACE:     return self.wait_multi(self.channels.values(), timeout=timeout)
(nova.rpc.common): TRACE:   File "/usr/local/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 56, in wait_multi
(nova.rpc.common): TRACE:     chanmap.keys(), allowed_methods, timeout=timeout)
(nova.rpc.common): TRACE:   File "/usr/local/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 104, in _wait_multiple
(nova.rpc.common): TRACE:     channel, method_sig, args, content = self.read_timeout(timeout)
(nova.rpc.common): TRACE:   File "/usr/local/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 86, in read_timeout
(nova.rpc.common): TRACE:     return self.method_reader.read_method()
(nova.rpc.common): TRACE:   File "/usr/local/lib/python2.7/dist-packages/amqplib/client_0_8/method_framing.py", line 215, in read_method
(nova.rpc.common): TRACE:     raise m
(nova.rpc.common): TRACE: timeout: timed out
(nova.rpc.common): TRACE:
2012-03-26 13:45:36 ERROR nova.api.openstack [req-161614ff-cf70-4aba-a8f2-9116924eec7b 8e5902533835427b8f812ec60c909546 a07fd73258684caaa86edcaabd7a16b0] Caught error: Timeout while waiting on RPC response.
(nova.api.openstack): TRACE: Traceback (most recent call last):
(nova.api.openstack): TRACE:   File "/usr/local/lib/python2.7/dist-packages/nova-2012.1-py2.7.egg/nova/api/openstack/__init__.py", line 41, in __call__
(nova.api.openstack): TRACE:     return req.get_response(self.application)
(nova.api.openstack): TRACE:   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1053, in get_response
(nova.api.openstack): TRACE:     application, catch_exc_info=False)
(nova.api.openstack): TRACE:   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1022, in call_application
(nova.api.openstack): TRACE:     app_iter = application(self.environ, start_response)
(nova.api.openstack): TRACE:   File "/usr/local/lib/python2.7/dist-packages/keystone-2012.1-py2.7.egg/keystone/middleware/auth_token.py", line 176, in __call__
(nova.api.openstack): TRACE:     return self.app(env, start_response)
(nova.api.openstack): TRACE:   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
(nova.api.openstack): TRACE:     return resp(environ, start_response)
(nova.api.openstack): TRACE:   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
(nova.api.openstack): TRACE:     return resp(environ, start_response)
(nova.api.openstack): TRACE:   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
(nova.api.openstack): TRACE:     return resp(environ, start_response)
(nova.api.openstack): TRACE:   File "/usr/local/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
(nova.api.openstack): TRACE:     response = self.app(environ, start_response)
(nova.api.openstack): TRACE:   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
(nova.api.openstack): TRACE:     return resp(environ, start_response)
(nova.api.openstack): TRACE:   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 147, in __call__
(nova.api.openstack): TRACE:     resp = self.call_func(req, *args, **self.kwargs)
(nova.api.openstack): TRACE:   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 208, in call_func
(nova.api.openstack): TRACE:     return self.func(req, *args, **kwargs)
(nova.api.openstack): TRACE:   File "/usr/local/lib/python2.7/dist-packages/nova-2012.1-py2.7.egg/nova/api/openstack/wsgi.py", line 803, in __call__
(nova.api.openstack): TRACE:     content_type, body, accept)
(nova.api.openstack): TRACE:   File "/usr/local/lib/python2.7/dist-packages/nova-2012.1-py2.7.egg/nova/api/openstack/wsgi.py", line 851, in _process_stack
(nova.api.openstack): TRACE:     action_result = self.dispatch(meth, request, action_args)
(nova.api.openstack): TRACE:   File "/usr/local/lib/python2.7/dist-packages/nova-2012.1-py2.7.egg/nova/api/openstack/wsgi.py", line 926, in dispatch
(nova.api.openstack): TRACE:     return method(req=request, **action_args)
(nova.api.openstack): TRACE:   File "/usr/local/lib/python2.7/dist-packages/nova-2012.1-py2.7.egg/nova/api/openstack/compute/contrib/certificates.py", line 79, in show
(nova.api.openstack): TRACE:     "args": {"project_id": context.project_id}})
(nova.api.openstack): TRACE:   File "/usr/local/lib/python2.7/dist-packages/nova-2012.1-py2.7.egg/nova/rpc/__init__.py", line 68, in call
(nova.api.openstack): TRACE:     return _get_impl().call(context, topic, msg, timeout)
(nova.api.openstack): TRACE:   File "/usr/local/lib/python2.7/dist-packages/nova-2012.1-py2.7.egg/nova/rpc/impl_kombu.py", line 674, in call
(nova.api.openstack): TRACE:     return rpc_amqp.call(context, topic, msg, timeout, Connection.pool)
(nova.api.openstack): TRACE:   File "/usr/local/lib/python2.7/dist-packages/nova-2012.1-py2.7.egg/nova/rpc/amqp.py", line 338, in call
(nova.api.openstack): TRACE:     rv = list(rv)
(nova.api.openstack): TRACE:   File "/usr/local/lib/python2.7/dist-packages/nova-2012.1-py2.7.egg/nova/rpc/amqp.py", line 299, in __iter__
(nova.api.openstack): TRACE:     self._iterator.next()
(nova.api.openstack): TRACE:   File "/usr/local/lib/python2.7/dist-packages/nova-2012.1-py2.7.egg/nova/rpc/impl_kombu.py", line 572, in iterconsume
(nova.api.openstack): TRACE:     yield self.ensure(_error_callback, _consume)
(nova.api.openstack): TRACE:   File "/usr/local/lib/python2.7/dist-packages/nova-2012.1-py2.7.egg/nova/rpc/impl_kombu.py", line 503, in ensure
(nova.api.openstack): TRACE:     error_callback(e)
(nova.api.openstack): TRACE:   File "/usr/local/lib/python2.7/dist-packages/nova-2012.1-py2.7.egg/nova/rpc/impl_kombu.py", line 553, in _error_callback
(nova.api.openstack): TRACE:     raise rpc_common.Timeout()
(nova.api.openstack): TRACE: Timeout: Timeout while waiting on RPC response.
(nova.api.openstack): TRACE:
2012-03-26 13:45:36 INFO nova.api.openstack [req-161614ff-cf70-4aba-a8f2-9116924eec7b 8e5902533835427b8f812ec60c909546 a07fd73258684caaa86edcaabd7a16b0] http://192.168.122.101:8774/v1.1/a07fd73258684caaa86edcaabd7a16b0/os-certificates/root returned with HTTP 500

Follow ups

Re: essex rc1: rabbit time out when runing nova x509-get-root-cert
From: Julien Danjou, 2012-03-26