| Thread Previous • Date Previous • Date Next • Thread Next |
From: John Griffith <john.griffith@xxxxxxxxxxxxx<mailto:john.griffith@xxxxxxxxxxxxx>> Date: Monday, November 12, 2012 7:47 PM To: Ahmed Al-Mehdi <ahmed@xxxxxxxxxx<mailto:ahmed@xxxxxxxxxx>> Cc: Jian Hua Geng <gengjh@xxxxxxxxxx<mailto:gengjh@xxxxxxxxxx>>, "openstack-bounces+gengjh=cn.ibm.com@xxxxxxxxxxxxxxxxxxx<mailto:openstack-bounces+gengjh=cn.ibm.com@xxxxxxxxxxxxxxxxxxx>" <openstack-bounces+gengjh=cn.ibm.com@xxxxxxxxxxxxxxxxxxx<mailto:openstack-bounces+gengjh=cn.ibm.com@xxxxxxxxxxxxxxxxxxx>>, "openstack@xxxxxxxxxxxxxxxxxxx<mailto:openstack@xxxxxxxxxxxxxxxxxxx>" <openstack@xxxxxxxxxxxxxxxxxxx<mailto:openstack@xxxxxxxxxxxxxxxxxxx>> Subject: Re: [Openstack] Cannot start "nova-api" service On Mon, Nov 12, 2012 at 7:34 PM, Ahmed Al-Mehdi <ahmed@xxxxxxxxxx<mailto:ahmed@xxxxxxxxxx>> wrote: From: John Griffith <john.griffith@xxxxxxxxxxxxx<mailto:john.griffith@xxxxxxxxxxxxx>> Date: Monday, November 12, 2012 7:17 PM To: Jian Hua Geng <gengjh@xxxxxxxxxx<mailto:gengjh@xxxxxxxxxx>> Cc: Ahmed Al-Mehdi <ahmed@xxxxxxxxxx<mailto:ahmed@xxxxxxxxxx>>, "openstack-bounces+gengjh=cn.ibm.com@xxxxxxxxxxxxxxxxxxx<mailto:openstack-bounces+gengjh=cn.ibm.com@xxxxxxxxxxxxxxxxxxx>" <openstack-bounces+gengjh=cn.ibm.com@xxxxxxxxxxxxxxxxxxx<mailto:openstack-bounces+gengjh=cn.ibm.com@xxxxxxxxxxxxxxxxxxx>>, "openstack@xxxxxxxxxxxxxxxxxxx<mailto:openstack@xxxxxxxxxxxxxxxxxxx>" <openstack@xxxxxxxxxxxxxxxxxxx<mailto:openstack@xxxxxxxxxxxxxxxxxxx>> Subject: Re: [Openstack] Cannot start "nova-api" service On Mon, Nov 12, 2012 at 6:57 PM, Jian Hua Geng <gengjh@xxxxxxxxxx<mailto:gengjh@xxxxxxxxxx>> wrote: By default both cinder and nova-api are listening on the same port 8776 (this should be a bug I think), you can try to change the default value in the cinder.conf like: osapi_volume_listen_port = 8777 if you are running the cinder and nova-api on the same machine. -------------------------------------------------- Best regard, David Geng -------------------------------------------------- [cid:1__=C7BBF026DF9980448f9e8a93df938@xxxxxxxxxx]Ahmed Al-Mehdi ---11/13/2012 09:32:21 AM---Ahmed Al-Mehdi <ahmed@xxxxxxxxxx<mailto:ahmed@xxxxxxxxxx>> Ahmed Al-Mehdi <ahmed@xxxxxxxxxx<mailto:ahmed@xxxxxxxxxx>> Sent by: openstack-bounces+gengjh=cn.ibm.com@xxxxxxxxxxxxxxxxxxx<mailto:cn.ibm.com@xxxxxxxxxxxxxxxxxxx> 11/13/2012 09:32 AM To Vishvananda Ishaya <vishvananda@xxxxxxxxx<mailto:vishvananda@xxxxxxxxx>>, cc "openstack@xxxxxxxxxxxxxxxxxxx<mailto:openstack@xxxxxxxxxxxxxxxxxxx>" <openstack@xxxxxxxxxxxxxxxxxxx<mailto:openstack@xxxxxxxxxxxxxxxxxxx>> Subject Re: [Openstack] Cannot start "nova-api" service Hello, Can someone please help me with a nova-api issue. After install all the nova services, all seem to be running fine, except for nova-api. I even reboot my controller node, no luck. After reboot all services are running, except nova-api. When I manually start nova-api, nova-api crashes with the following error "error: [Errno 98] Address already in use". I installed nova-volume earlier during the install process, but later on installed cinder, and made the necessary modifications (as far as I can tell) to nova-api.conf to use cinder for block storage. Should I uninstall nova-volume? 2012-11-12 14:46:24 INFO keystone.middleware.auth_token [-] Starting keystone auth_token middleware 2012-11-12 14:46:24 INFO keystone.middleware.auth_token [-] Using /var/lib/nova/keystone-signing as cache directory for signing certificate 2012-11-12 14:46:24 CRITICAL nova [-] [Errno 98] Address already in use 2012-11-12 14:46:24 TRACE nova Traceback (most recent call last): 2012-11-12 14:46:24 TRACE nova File "/usr/bin/nova-api", line 50, in <module> 2012-11-12 14:46:24 TRACE nova server = service.WSGIService(api) 2012-11-12 14:46:24 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 584, in __init__ 2012-11-12 14:46:24 TRACE nova port=self.port) 2012-11-12 14:46:24 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/wsgi.py", line 72, in __init__ 2012-11-12 14:46:24 TRACE nova self._socket = eventlet.listen((host, port), backlog=backlog) 2012-11-12 14:46:24 TRACE nova File "/usr/lib/python2.7/dist-packages/eventlet/convenience.py", line 38, in listen 2012-11-12 14:46:24 TRACE nova sock.bind(addr) 2012-11-12 14:46:24 TRACE nova File "/usr/lib/python2.7/socket.py", line 224, in meth 2012-11-12 14:46:24 TRACE nova return getattr(self._sock,name)(*args) 2012-11-12 14:46:24 TRACE nova error: [Errno 98] Address already in use 2012-11-12 14:46:24 TRACE nova 2012-11-12 14:46:24 INFO nova.service [-] Parent process has died unexpectedly, exiting 2012-11-12 14:46:24 INFO nova.service [-] Parent process has died unexpectedly, exiting 2012-11-12 14:46:24 INFO nova.wsgi [-] Stopping WSGI server. 2012-11-12 14:46:24 INFO nova.wsgi [-] Stopping WSGI server. Would highly appreciate any pointers to understanding or resolving the issue. Regards, Ahmed. From: Ahmed Al-Mehdi <ahmed@xxxxxxxxxx<mailto:ahmed@xxxxxxxxxx>> Date: Friday, November 9, 2012 12:45 AM To: Vishvananda Ishaya <vishvananda@xxxxxxxxx<mailto:vishvananda@xxxxxxxxx>> Cc: "openstack@xxxxxxxxxxxxxxxxxxx<mailto:openstack@xxxxxxxxxxxxxxxxxxx>" <openstack@xxxxxxxxxxxxxxxxxxx<mailto:openstack@xxxxxxxxxxxxxxxxxxx>> Subject: Re: [Openstack] Cannot start "nova-api" service From: Vishvananda Ishaya <vishvananda@xxxxxxxxx<mailto:vishvananda@xxxxxxxxx>> Date: Thursday, November 8, 2012 8:18 PM To: Ahmed Al-Mehdi <ahmed@xxxxxxxxxx<mailto:ahmed@xxxxxxxxxx>> Cc: "openstack@xxxxxxxxxxxxxxxxxxx<mailto:openstack@xxxxxxxxxxxxxxxxxxx>" <openstack@xxxxxxxxxxxxxxxxxxx<mailto:openstack@xxxxxxxxxxxxxxxxxxx>> Subject: Re: [Openstack] Cannot start "nova-api" service On Nov 8, 2012, at 7:01 PM, Ahmed Al-Mehdi <ahmed@xxxxxxxxxx<mailto:ahmed@xxxxxxxxxx>> wrote: Vish, I am running cinder-api. The following two lines are present in nova.conf. volume_api_class=nova.volume.cinder.API enabled_apis=ec2,osapi_compute,metadata Do I need to re-sync the db, or add any additional lines to nova.conf? No that is it. Are you sure a) you don't have another nova-api or nova-metadata or nova-api-os-compute process running? and b) that your nova.conf is being read properly? Vish As far as I can tell, no other nova-api, nova-metadata, nova-api-os-compute is running. If there another way to confirm besides running "ps aux". And how can I tell if nova.conf is being read properly? root@bodega:~# ps aux | grep nova nova 914 0.0 0.0 37952 1312 ? Ss 16:01 0:00 su -s /bin/sh -c exec nova-novncproxy --config-file=/etc/nova/nova.conf nova nova 916 0.0 0.2 122976 24108 ? S 16:01 0:01 /usr/bin/python /usr/bin/nova-novncproxy --config-file=/etc/nova/nova.conf nova 1235 0.0 0.0 37952 1312 ? Ss 16:01 0:00 su -s /bin/sh -c exec nova-cert --config-file=/etc/nova/nova.conf nova nova 1243 0.0 0.0 37952 1308 ? Ss 16:01 0:00 su -s /bin/sh -c exec nova-consoleauth --config-file=/etc/nova/nova.conf nova nova 1244 0.2 0.6 122996 51232 ? S 16:01 1:12 /usr/bin/python /usr/bin/nova-cert --config-file=/etc/nova/nova.conf nova 1249 0.2 0.6 122992 51252 ? S 16:01 1:13 /usr/bin/python /usr/bin/nova-consoleauth --config-file=/etc/nova/nova.conf nova 1252 0.0 0.0 37952 1312 ? Ss 16:01 0:00 su -s /bin/sh -c exec nova-network --config-file=/etc/nova/nova.conf nova nova 1255 0.0 0.0 37952 1308 ? Ss 16:01 0:00 su -s /bin/sh -c exec nova-scheduler --config-file=/etc/nova/nova.conf nova nova 1259 0.3 0.6 124964 53100 ? S 16:01 1:16 /usr/bin/python /usr/bin/nova-network --config-file=/etc/nova/nova.conf nova 1260 0.3 0.7 151856 59068 ? S 16:01 1:16 /usr/bin/python /usr/bin/nova-scheduler --config-file=/etc/nova/nova.conf root 3509 0.0 0.0 9388 920 pts/3 S+ 22:55 0:00 grep --color=auto nova root@bodega:~# root@bodega:~# root@bodega:~# ls -l /etc/nova/ total 32 -rw-r----- 1 nova nova 3588 Sep 25 17:48 api-paste.ini -rw-r-xr-x 1 nova nova 1329 Oct 20 19:16 logging.conf -rw-r----- 1 nova nova 2203 Nov 8 18:34 nova.conf -rw-r----- 1 root root 434 Nov 5 10:44 nova.conf.orig.ahmed -rw-r----- 1 nova nova 5181 Sep 25 17:48 policy.json -rw-r--r-- 1 root root 304 Sep 25 17:48 rootwrap.conf drwxr-xr-x 2 root root 4096 Nov 5 10:36 rootwrap.d root@bodega:~# Can you help me understand the following error message in the file: 2012-11-08 23:31:27 CRITICAL nova [-] [Errno 98] Address already in use By address, are we talking about tcp port number? If so, what is the port number? Thank you, Ahmed. _______________________________________________ Mailing list: https://launchpad.net/~openstack Post to : openstack@xxxxxxxxxxxxxxxxxxx<mailto:openstack@xxxxxxxxxxxxxxxxxxx> Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp _______________________________________________ Mailing list: https://launchpad.net/~openstack Post to : openstack@xxxxxxxxxxxxxxxxxxx<mailto:openstack@xxxxxxxxxxxxxxxxxxx> Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp Ahmed, Seems to me like you still have nova-volume configured. BTW, as mentioned earlier nova-volume and cinder-volume using the same port is NOT a bug, it's by design. Also there seems to be some confusion here, osapi_compute does NOT use 8776, it uses 8774 by default. Something that was pointed out earlier today is that the install document lists everything under the keystone ini heading. If you copied your nova.conf exactly like the doc then your entry for: enabled_apis=ec2,osapi_compute,metadata Is going to be ignored. Would you please provide your nova.conf and cinder.conf files (link to a pastebin perhaps), and we can verify this. John Hi John, When you say I still have nova-volume configured, do you mean it is still running on my controller node. I don't think nova-volume is running (from output of "ps aux | nova"), but I could be wrong. However, nova-volume is still configured in keystone. I copied the nova.conf and cinder.conf file onto paste bin - http://pastebin.com/xtpVKzs0. Thank you very much for your help. root@bodega:~/ahmed# keystone endpoint-list +----------------------------------+-----------+-------------------------------------------------+-------------------------------------------------+--------------------------------------------+ | id | region | publicurl | internalurl | adminurl | +----------------------------------+-----------+-------------------------------------------------+-------------------------------------------------+--------------------------------------------+ | 1753b8533e474cf0934cd2eb1d23be45 | RegionOne | http://10.176.20.158:8888/v1/AUTH_%(tenant_id)s | http://10.176.20.158:8888/v1/AUTH_%(tenant_id)s | http://10.176.20.158:8888/v1 | | 35c19563fce34c04a20cc82952b096b5 | RegionOne | http://10.176.20.158:8773/services/Cloud | http://10.176.20.158:8773/services/Cloud | http://10.176.20.158:8773/services/Admin | | 36f5a5c3021a485db6b900aee9d7520c | RegionOne | http://10.176.20.158:9292/v1 | http://10.176.20.158:9292/v1 | http://10.176.20.158:9292/v1 | | 7fdf2e50d29a454897f3c5d395a8326f | RegionOne | http://10.176.20.158:8774/v2/%(tenant_id)s | http://10.176.20.158:8774/v2/%(tenant_id)s | http://10.176.20.158:8774/v2/%(tenant_id)s | | 9b1ade95c694401cb61362daf281713b | RegionOne | http://10.176.20.158:5000/v2.0 | http://10.176.20.158:5000/v2.0 | http://10.176.20.158:35357/v2.0 | | d728f31d9745467aaf53eeeba633ffe4 | RegionOne | http://10.176.20.158:8776/v1/%(tenant_id)s | http://10.176.20.158:8776/v1/%(tenant_id)s | http://10.176.20.158:8776/v1/%(tenant_id)s | | e298efba7f0148819c400a26e7f6f448 | RegionOne | http://10.176.20.158:8776/v1/%(tenant_id)s | http://10.176.20.158:8776/v1/%(tenant_id)s | http://10.176.20.158:8776/v1/%(tenant_id)s | +----------------------------------+-----------+-------------------------------------------------+-------------------------------------------------+--------------------------------------------+ root@bodega:~/ahmed# keystone service-list +----------------------------------+----------+--------------+---------------------------+ | id | name | type | description | +----------------------------------+----------+--------------+---------------------------+ | 083662fed26d490b88172a3aa638107a | volume | volume | Nova Volume Service | | 3224949d951a4fb3b45adb5778caebfe | cinder | volume | Cinder Volume Service | | 55f097212ab948e5a2bf13e47ac1be9c | ec2 | ec2 | EC2 Compatibility Layer | | 611a8d8380de4671863c2cd59a4d5bd8 | glance | image | Glance Image Service | | 90eedca2364b4a5bba477be31738c052 | keystone | identity | Keystone Identity Service | | a6552ffaa4904ec09ef399d71dd5e18f | swift | object-store | Object Storage Service | | f0f5c38f832f4584b93c562d1d756fa3 | nova | compute | Nova Compute Service | +----------------------------------+----------+--------------+---------------------------+ root@bodega:~/ahmed# Thank you, Ahmed. Ahmed, This entry in your nova.conf is the issue: [keystone_authtoken] Remove that line and restart and you should be correct. What's happening in your case is everything after this entry that is NOT keystone_authtoken related is basically being ignored and the defaults are picked up instead. John John, Thank you very much for your help. Is there a document that explains the syntax of the conf file structure, or is that a bug in how the conf file is parsed by nova? After making the changes and rebooting the controller node, I see nova-api up and running. However, I now see four instances of "nova-api" running. Is that an issue? root@bodega:~# ps aux | grep nova nova 979 0.0 0.0 37952 1312 ? Ss 18:58 0:00 su -s /bin/sh -c exec nova-novncproxy --config-file=/etc/nova/nova.conf nova nova 981 0.0 0.2 122968 24096 ? S 18:58 0:00 /usr/bin/python /usr/bin/nova-novncproxy --config-file=/etc/nova/nova.conf nova 1301 0.0 0.0 37952 1308 ? Ss 18:59 0:00 su -s /bin/sh -c exec nova-api --config-file=/etc/nova/nova.conf nova nova 1307 0.5 0.7 135476 57388 ? S 18:59 0:01 /usr/bin/python /usr/bin/nova-api --config-file=/etc/nova/nova.conf nova 1517 0.0 0.2 95780 22816 ? S 18:59 0:00 /usr/bin/python /usr/bin/nova-api --config-file=/etc/nova/nova.conf nova 1636 0.0 0.6 134352 53060 ? S 18:59 0:00 /usr/bin/python /usr/bin/nova-api --config-file=/etc/nova/nova.conf nova 1639 0.0 0.6 135476 54332 ? S 18:59 0:00 /usr/bin/python /usr/bin/nova-api --config-file=/etc/nova/nova.conf nova 5760 0.0 0.0 37952 1308 ? Ss 19:02 0:00 su -s /bin/sh -c exec nova-consoleauth --config-file=/etc/nova/nova.conf nova nova 5761 0.0 0.3 88192 28876 ? R 19:02 0:00 /usr/bin/python /usr/bin/nova-consoleauth --config-file=/etc/nova/nova.conf root 5767 0.0 0.0 9388 920 pts/2 S+ 19:02 0:00 grep --color=auto nova root@bodega:~# Prior to the reboot (before I fixed nova.conf), I see nova-cert, nova-consoleauth, nova-network, nova-scheduler running (automatically started upon boot), which are not automatically started after the nova.conf file change. Is that an issue? Or is that due to the default setting being enabled by the issue in nova.conf, which triggers the launch of these additional services? root@bodega:~/ahmed# ps aux | grep nova nova 958 0.0 0.0 37952 1312 ? Ss Nov08 0:00 su -s /bin/sh -c exec nova-novncproxy --config-file=/etc/nova/nova.conf nova nova 959 0.0 0.2 122972 24104 ? S Nov08 0:15 /usr/bin/python /usr/bin/nova-novncproxy --config-file=/etc/nova/nova.conf nova 1276 0.0 0.0 37952 1308 ? Ss Nov08 0:00 su -s /bin/sh -c exec nova-cert --config-file=/etc/nova/nova.conf nova nova 1287 0.2 0.6 122988 51232 ? S Nov08 15:40 /usr/bin/python /usr/bin/nova-cert --config-file=/etc/nova/nova.conf nova 1288 0.0 0.0 37952 1312 ? Ss Nov08 0:00 su -s /bin/sh -c exec nova-consoleauth --config-file=/etc/nova/nova.conf nova nova 1290 0.0 0.0 37952 1312 ? Ss Nov08 0:00 su -s /bin/sh -c exec nova-network --config-file=/etc/nova/nova.conf nova nova 1293 0.2 0.6 123020 51252 ? S Nov08 15:01 /usr/bin/python /usr/bin/nova-consoleauth --config-file=/etc/nova/nova.conf nova 1294 0.3 0.6 124964 53320 ? S Nov08 16:13 /usr/bin/python /usr/bin/nova-network --config-file=/etc/nova/nova.conf nova 1303 0.0 0.0 37952 1312 ? Ss Nov08 0:00 su -s /bin/sh -c exec nova-scheduler --config-file=/etc/nova/nova.conf nova nova 1306 0.3 0.7 151960 59428 ? S Nov08 16:14 /usr/bin/python /usr/bin/nova-scheduler --config-file=/etc/nova/nova.conf Thank you, Ahmed.
Attachment:
ecblank.gif
Description: ecblank.gif
Attachment:
graycol.gif
Description: graycol.gif
| Thread Previous • Date Previous • Date Next • Thread Next |
