← Back to team overview

openstack team mailing list archive

Re: Cannot start "nova-api" service

 


From: John Griffith <john.griffith@xxxxxxxxxxxxx<mailto:john.griffith@xxxxxxxxxxxxx>>
Date: Monday, November 12, 2012 7:47 PM
To: Ahmed Al-Mehdi <ahmed@xxxxxxxxxx<mailto:ahmed@xxxxxxxxxx>>
Cc: Jian Hua Geng <gengjh@xxxxxxxxxx<mailto:gengjh@xxxxxxxxxx>>, "openstack-bounces+gengjh=cn.ibm.com@xxxxxxxxxxxxxxxxxxx<mailto:openstack-bounces+gengjh=cn.ibm.com@xxxxxxxxxxxxxxxxxxx>" <openstack-bounces+gengjh=cn.ibm.com@xxxxxxxxxxxxxxxxxxx<mailto:openstack-bounces+gengjh=cn.ibm.com@xxxxxxxxxxxxxxxxxxx>>, "openstack@xxxxxxxxxxxxxxxxxxx<mailto:openstack@xxxxxxxxxxxxxxxxxxx>" <openstack@xxxxxxxxxxxxxxxxxxx<mailto:openstack@xxxxxxxxxxxxxxxxxxx>>
Subject: Re: [Openstack] Cannot start "nova-api" service



On Mon, Nov 12, 2012 at 7:34 PM, Ahmed Al-Mehdi <ahmed@xxxxxxxxxx<mailto:ahmed@xxxxxxxxxx>> wrote:


From: John Griffith <john.griffith@xxxxxxxxxxxxx<mailto:john.griffith@xxxxxxxxxxxxx>>
Date: Monday, November 12, 2012 7:17 PM
To: Jian Hua Geng <gengjh@xxxxxxxxxx<mailto:gengjh@xxxxxxxxxx>>
Cc: Ahmed Al-Mehdi <ahmed@xxxxxxxxxx<mailto:ahmed@xxxxxxxxxx>>, "openstack-bounces+gengjh=cn.ibm.com@xxxxxxxxxxxxxxxxxxx<mailto:openstack-bounces+gengjh=cn.ibm.com@xxxxxxxxxxxxxxxxxxx>" <openstack-bounces+gengjh=cn.ibm.com@xxxxxxxxxxxxxxxxxxx<mailto:openstack-bounces+gengjh=cn.ibm.com@xxxxxxxxxxxxxxxxxxx>>, "openstack@xxxxxxxxxxxxxxxxxxx<mailto:openstack@xxxxxxxxxxxxxxxxxxx>" <openstack@xxxxxxxxxxxxxxxxxxx<mailto:openstack@xxxxxxxxxxxxxxxxxxx>>

Subject: Re: [Openstack] Cannot start "nova-api" service



On Mon, Nov 12, 2012 at 6:57 PM, Jian Hua Geng <gengjh@xxxxxxxxxx<mailto:gengjh@xxxxxxxxxx>> wrote:

By default both cinder and nova-api are listening on the same port 8776 (this should be a bug I think), you can try to change the default value in the cinder.conf like: osapi_volume_listen_port = 8777 if you are running the cinder and nova-api on the same machine.

--------------------------------------------------
Best regard,
David Geng

--------------------------------------------------

[cid:1__=C7BBF026DF9980448f9e8a93df938@xxxxxxxxxx]Ahmed Al-Mehdi ---11/13/2012 09:32:21 AM---Ahmed Al-Mehdi <ahmed@xxxxxxxxxx<mailto:ahmed@xxxxxxxxxx>>


Ahmed Al-Mehdi <ahmed@xxxxxxxxxx<mailto:ahmed@xxxxxxxxxx>>
Sent by: openstack-bounces+gengjh=cn.ibm.com@xxxxxxxxxxxxxxxxxxx<mailto:cn.ibm.com@xxxxxxxxxxxxxxxxxxx>

11/13/2012 09:32 AM



To


Vishvananda Ishaya <vishvananda@xxxxxxxxx<mailto:vishvananda@xxxxxxxxx>>,



cc


"openstack@xxxxxxxxxxxxxxxxxxx<mailto:openstack@xxxxxxxxxxxxxxxxxxx>" <openstack@xxxxxxxxxxxxxxxxxxx<mailto:openstack@xxxxxxxxxxxxxxxxxxx>>



Subject



Re: [Openstack] Cannot start "nova-api" service





Hello,

Can someone please help me with a nova-api issue.  After install all the nova services, all seem to be running fine, except for nova-api.  I even reboot my controller node, no luck.  After reboot all services are running, except nova-api.  When I manually start nova-api, nova-api crashes with the following error "error: [Errno 98] Address already in use".  I installed nova-volume earlier during the install process, but later on installed cinder, and made the necessary modifications (as far as I can tell) to nova-api.conf to use cinder for block storage.  Should I uninstall nova-volume?

2012-11-12 14:46:24 INFO keystone.middleware.auth_token [-] Starting keystone auth_token middleware
2012-11-12 14:46:24 INFO keystone.middleware.auth_token [-] Using /var/lib/nova/keystone-signing as cache directory for signing certificate
2012-11-12 14:46:24 CRITICAL nova [-] [Errno 98] Address already in use
2012-11-12 14:46:24 TRACE nova Traceback (most recent call last):
2012-11-12 14:46:24 TRACE nova   File "/usr/bin/nova-api", line 50, in <module>
2012-11-12 14:46:24 TRACE nova     server = service.WSGIService(api)
2012-11-12 14:46:24 TRACE nova   File "/usr/lib/python2.7/dist-packages/nova/service.py", line 584, in __init__
2012-11-12 14:46:24 TRACE nova     port=self.port)
2012-11-12 14:46:24 TRACE nova   File "/usr/lib/python2.7/dist-packages/nova/wsgi.py", line 72, in __init__
2012-11-12 14:46:24 TRACE nova     self._socket = eventlet.listen((host, port), backlog=backlog)
2012-11-12 14:46:24 TRACE nova   File "/usr/lib/python2.7/dist-packages/eventlet/convenience.py", line 38, in listen
2012-11-12 14:46:24 TRACE nova     sock.bind(addr)
2012-11-12 14:46:24 TRACE nova   File "/usr/lib/python2.7/socket.py", line 224, in meth
2012-11-12 14:46:24 TRACE nova     return getattr(self._sock,name)(*args)
2012-11-12 14:46:24 TRACE nova error: [Errno 98] Address already in use
2012-11-12 14:46:24 TRACE nova
2012-11-12 14:46:24 INFO nova.service [-] Parent process has died unexpectedly, exiting
2012-11-12 14:46:24 INFO nova.service [-] Parent process has died unexpectedly, exiting
2012-11-12 14:46:24 INFO nova.wsgi [-] Stopping WSGI server.
2012-11-12 14:46:24 INFO nova.wsgi [-] Stopping WSGI server.

Would highly appreciate any pointers to understanding or resolving the issue.

Regards,
Ahmed.


From: Ahmed Al-Mehdi <ahmed@xxxxxxxxxx<mailto:ahmed@xxxxxxxxxx>>
Date: Friday, November 9, 2012 12:45 AM
To: Vishvananda Ishaya <vishvananda@xxxxxxxxx<mailto:vishvananda@xxxxxxxxx>>
Cc: "openstack@xxxxxxxxxxxxxxxxxxx<mailto:openstack@xxxxxxxxxxxxxxxxxxx>" <openstack@xxxxxxxxxxxxxxxxxxx<mailto:openstack@xxxxxxxxxxxxxxxxxxx>>
Subject: Re: [Openstack] Cannot start "nova-api" service


From: Vishvananda Ishaya <vishvananda@xxxxxxxxx<mailto:vishvananda@xxxxxxxxx>>
Date: Thursday, November 8, 2012 8:18 PM
To: Ahmed Al-Mehdi <ahmed@xxxxxxxxxx<mailto:ahmed@xxxxxxxxxx>>
Cc: "openstack@xxxxxxxxxxxxxxxxxxx<mailto:openstack@xxxxxxxxxxxxxxxxxxx>" <openstack@xxxxxxxxxxxxxxxxxxx<mailto:openstack@xxxxxxxxxxxxxxxxxxx>>
Subject: Re: [Openstack] Cannot start "nova-api" service

On Nov 8, 2012, at 7:01 PM, Ahmed Al-Mehdi <ahmed@xxxxxxxxxx<mailto:ahmed@xxxxxxxxxx>> wrote:
Vish,

I am running cinder-api.   The following two lines are present in nova.conf.

volume_api_class=nova.volume.cinder.API
enabled_apis=ec2,osapi_compute,metadata

Do I need to re-sync the db, or add any additional lines to nova.conf?

No that is it. Are you sure a) you don't have another nova-api or nova-metadata or nova-api-os-compute process running? and b) that your nova.conf is being read properly?

Vish

As far as I can tell, no other nova-api, nova-metadata, nova-api-os-compute is running.   If there another way to confirm besides running "ps aux".  And how can I tell if nova.conf is being read properly?

root@bodega:~# ps aux | grep nova
nova       914  0.0  0.0  37952  1312 ?        Ss   16:01   0:00 su -s /bin/sh -c exec nova-novncproxy --config-file=/etc/nova/nova.conf nova
nova       916  0.0  0.2 122976 24108 ?        S    16:01   0:01 /usr/bin/python /usr/bin/nova-novncproxy --config-file=/etc/nova/nova.conf
nova      1235  0.0  0.0  37952  1312 ?        Ss   16:01   0:00 su -s /bin/sh -c exec nova-cert --config-file=/etc/nova/nova.conf nova
nova      1243  0.0  0.0  37952  1308 ?        Ss   16:01   0:00 su -s /bin/sh -c exec nova-consoleauth --config-file=/etc/nova/nova.conf nova
nova      1244  0.2  0.6 122996 51232 ?        S    16:01   1:12 /usr/bin/python /usr/bin/nova-cert --config-file=/etc/nova/nova.conf
nova      1249  0.2  0.6 122992 51252 ?        S    16:01   1:13 /usr/bin/python /usr/bin/nova-consoleauth --config-file=/etc/nova/nova.conf
nova      1252  0.0  0.0  37952  1312 ?        Ss   16:01   0:00 su -s /bin/sh -c exec nova-network --config-file=/etc/nova/nova.conf nova
nova      1255  0.0  0.0  37952  1308 ?        Ss   16:01   0:00 su -s /bin/sh -c exec nova-scheduler --config-file=/etc/nova/nova.conf nova
nova      1259  0.3  0.6 124964 53100 ?        S    16:01   1:16 /usr/bin/python /usr/bin/nova-network --config-file=/etc/nova/nova.conf
nova      1260  0.3  0.7 151856 59068 ?        S    16:01   1:16 /usr/bin/python /usr/bin/nova-scheduler --config-file=/etc/nova/nova.conf
root      3509  0.0  0.0   9388   920 pts/3    S+   22:55   0:00 grep --color=auto nova
root@bodega:~#
root@bodega:~#
root@bodega:~# ls -l /etc/nova/
total 32
-rw-r----- 1 nova nova 3588 Sep 25 17:48 api-paste.ini
-rw-r-xr-x 1 nova nova 1329 Oct 20 19:16 logging.conf
-rw-r----- 1 nova nova 2203 Nov  8 18:34 nova.conf
-rw-r----- 1 root root  434 Nov  5 10:44 nova.conf.orig.ahmed
-rw-r----- 1 nova nova 5181 Sep 25 17:48 policy.json
-rw-r--r-- 1 root root  304 Sep 25 17:48 rootwrap.conf
drwxr-xr-x 2 root root 4096 Nov  5 10:36 rootwrap.d
root@bodega:~#


Can you help me understand the following error message in the file:

2012-11-08 23:31:27 CRITICAL nova [-] [Errno 98] Address already in use

By address, are we talking about tcp port number? If so, what is the port number?

Thank you,
Ahmed.

_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to     : openstack@xxxxxxxxxxxxxxxxxxx<mailto:openstack@xxxxxxxxxxxxxxxxxxx>
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp


_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to     : openstack@xxxxxxxxxxxxxxxxxxx<mailto:openstack@xxxxxxxxxxxxxxxxxxx>
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Ahmed,

Seems to me like you still have nova-volume configured.  BTW, as mentioned earlier nova-volume and cinder-volume using the same port is NOT a bug, it's by design. Also there seems to be some confusion here, osapi_compute does NOT use 8776, it uses 8774 by default.

Something that was pointed out earlier today is that the install document lists everything under the keystone ini heading.  If you copied your nova.conf exactly like the doc then your entry for: enabled_apis=ec2,osapi_compute,metadata  Is going to be ignored.

Would you please provide your nova.conf and cinder.conf files (link to a pastebin perhaps), and we can verify this.

John

Hi John,

When you say I still have nova-volume configured, do you mean it is still running on my controller node.  I don't think nova-volume is running (from output of "ps aux | nova"), but I could be wrong.  However, nova-volume is still configured in keystone.  I copied the nova.conf and cinder.conf file onto paste bin - http://pastebin.com/xtpVKzs0.  Thank you very much for your help.

root@bodega:~/ahmed# keystone endpoint-list
+----------------------------------+-----------+-------------------------------------------------+-------------------------------------------------+--------------------------------------------+
|                id                |   region  |                    publicurl                    |                   internalurl                   |                  adminurl                  |
+----------------------------------+-----------+-------------------------------------------------+-------------------------------------------------+--------------------------------------------+
| 1753b8533e474cf0934cd2eb1d23be45 | RegionOne | http://10.176.20.158:8888/v1/AUTH_%(tenant_id)s | http://10.176.20.158:8888/v1/AUTH_%(tenant_id)s |        http://10.176.20.158:8888/v1        |
| 35c19563fce34c04a20cc82952b096b5 | RegionOne |     http://10.176.20.158:8773/services/Cloud    |     http://10.176.20.158:8773/services/Cloud    |  http://10.176.20.158:8773/services/Admin  |
| 36f5a5c3021a485db6b900aee9d7520c | RegionOne |           http://10.176.20.158:9292/v1          |           http://10.176.20.158:9292/v1          |        http://10.176.20.158:9292/v1        |
| 7fdf2e50d29a454897f3c5d395a8326f | RegionOne |    http://10.176.20.158:8774/v2/%(tenant_id)s   |    http://10.176.20.158:8774/v2/%(tenant_id)s   | http://10.176.20.158:8774/v2/%(tenant_id)s |
| 9b1ade95c694401cb61362daf281713b | RegionOne |          http://10.176.20.158:5000/v2.0         |          http://10.176.20.158:5000/v2.0         |      http://10.176.20.158:35357/v2.0       |
| d728f31d9745467aaf53eeeba633ffe4 | RegionOne |    http://10.176.20.158:8776/v1/%(tenant_id)s   |    http://10.176.20.158:8776/v1/%(tenant_id)s   | http://10.176.20.158:8776/v1/%(tenant_id)s |
| e298efba7f0148819c400a26e7f6f448 | RegionOne |    http://10.176.20.158:8776/v1/%(tenant_id)s   |    http://10.176.20.158:8776/v1/%(tenant_id)s   | http://10.176.20.158:8776/v1/%(tenant_id)s |
+----------------------------------+-----------+-------------------------------------------------+-------------------------------------------------+--------------------------------------------+
root@bodega:~/ahmed# keystone service-list
+----------------------------------+----------+--------------+---------------------------+
|                id                |   name   |     type     |        description        |
+----------------------------------+----------+--------------+---------------------------+
| 083662fed26d490b88172a3aa638107a |  volume  |    volume    |    Nova Volume Service    |
| 3224949d951a4fb3b45adb5778caebfe |  cinder  |    volume    |   Cinder Volume Service   |
| 55f097212ab948e5a2bf13e47ac1be9c |   ec2    |     ec2      |  EC2 Compatibility Layer  |
| 611a8d8380de4671863c2cd59a4d5bd8 |  glance  |    image     |    Glance Image Service   |
| 90eedca2364b4a5bba477be31738c052 | keystone |   identity   | Keystone Identity Service |
| a6552ffaa4904ec09ef399d71dd5e18f |  swift   | object-store |   Object Storage Service  |
| f0f5c38f832f4584b93c562d1d756fa3 |   nova   |   compute    |    Nova Compute Service   |
+----------------------------------+----------+--------------+---------------------------+
root@bodega:~/ahmed#


Thank you,
Ahmed.


Ahmed,

This entry in your nova.conf is the issue: [keystone_authtoken]

Remove that line and restart and you should be correct.  What's happening in your case is everything after this entry that is NOT keystone_authtoken related is basically being ignored and the defaults are picked up instead.

John

John,

Thank you very much for your help.   Is there a document that explains the syntax of the conf file structure, or is that a bug in how the conf file is parsed by nova?

After making the changes and rebooting the controller node, I see nova-api up and running.  However, I now see four instances of "nova-api" running.  Is that an issue?

root@bodega:~# ps aux | grep nova
nova       979  0.0  0.0  37952  1312 ?        Ss   18:58   0:00 su -s /bin/sh -c exec nova-novncproxy --config-file=/etc/nova/nova.conf nova
nova       981  0.0  0.2 122968 24096 ?        S    18:58   0:00 /usr/bin/python /usr/bin/nova-novncproxy --config-file=/etc/nova/nova.conf
nova      1301  0.0  0.0  37952  1308 ?        Ss   18:59   0:00 su -s /bin/sh -c exec nova-api --config-file=/etc/nova/nova.conf nova
nova      1307  0.5  0.7 135476 57388 ?        S    18:59   0:01 /usr/bin/python /usr/bin/nova-api --config-file=/etc/nova/nova.conf
nova      1517  0.0  0.2  95780 22816 ?        S    18:59   0:00 /usr/bin/python /usr/bin/nova-api --config-file=/etc/nova/nova.conf
nova      1636  0.0  0.6 134352 53060 ?        S    18:59   0:00 /usr/bin/python /usr/bin/nova-api --config-file=/etc/nova/nova.conf
nova      1639  0.0  0.6 135476 54332 ?        S    18:59   0:00 /usr/bin/python /usr/bin/nova-api --config-file=/etc/nova/nova.conf
nova      5760  0.0  0.0  37952  1308 ?        Ss   19:02   0:00 su -s /bin/sh -c exec nova-consoleauth --config-file=/etc/nova/nova.conf nova
nova      5761  0.0  0.3  88192 28876 ?        R    19:02   0:00 /usr/bin/python /usr/bin/nova-consoleauth --config-file=/etc/nova/nova.conf
root      5767  0.0  0.0   9388   920 pts/2    S+   19:02   0:00 grep --color=auto nova
root@bodega:~#


Prior to the reboot (before I fixed nova.conf),  I see nova-cert, nova-consoleauth, nova-network, nova-scheduler running (automatically started upon boot), which are not automatically started after the nova.conf file change.  Is that an issue?  Or is that due to the default setting being enabled by the issue in nova.conf, which triggers the launch of these additional services?

root@bodega:~/ahmed# ps aux | grep nova
nova       958  0.0  0.0  37952  1312 ?        Ss   Nov08   0:00 su -s /bin/sh -c exec nova-novncproxy --config-file=/etc/nova/nova.conf nova
nova       959  0.0  0.2 122972 24104 ?        S    Nov08   0:15 /usr/bin/python /usr/bin/nova-novncproxy --config-file=/etc/nova/nova.conf
nova      1276  0.0  0.0  37952  1308 ?        Ss   Nov08   0:00 su -s /bin/sh -c exec nova-cert --config-file=/etc/nova/nova.conf nova
nova      1287  0.2  0.6 122988 51232 ?        S    Nov08  15:40 /usr/bin/python /usr/bin/nova-cert --config-file=/etc/nova/nova.conf
nova      1288  0.0  0.0  37952  1312 ?        Ss   Nov08   0:00 su -s /bin/sh -c exec nova-consoleauth --config-file=/etc/nova/nova.conf nova
nova      1290  0.0  0.0  37952  1312 ?        Ss   Nov08   0:00 su -s /bin/sh -c exec nova-network --config-file=/etc/nova/nova.conf nova
nova      1293  0.2  0.6 123020 51252 ?        S    Nov08  15:01 /usr/bin/python /usr/bin/nova-consoleauth --config-file=/etc/nova/nova.conf
nova      1294  0.3  0.6 124964 53320 ?        S    Nov08  16:13 /usr/bin/python /usr/bin/nova-network --config-file=/etc/nova/nova.conf
nova      1303  0.0  0.0  37952  1312 ?        Ss   Nov08   0:00 su -s /bin/sh -c exec nova-scheduler --config-file=/etc/nova/nova.conf nova
nova      1306  0.3  0.7 151960 59428 ?        S    Nov08  16:14 /usr/bin/python /usr/bin/nova-scheduler --config-file=/etc/nova/nova.conf


Thank you,
Ahmed.



Attachment: ecblank.gif
Description: ecblank.gif

Attachment: graycol.gif
Description: graycol.gif


References