yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1361180] Re: nova service disable/enable returns 500 on cell environment

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Alan Pevec <1361180@xxxxxxxxxxxxxxxxxx>
Date: Wed, 04 Mar 2015 22:37:57 -0000
Reply-to: Bug 1361180 <1361180@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Changed in: nova/juno
   Importance: Undecided => High

** Also affects: nova/icehouse
   Importance: Undecided
       Status: New

** Changed in: nova/icehouse
   Importance: Undecided => High

** Changed in: nova/icehouse
       Status: New => In Progress

** Changed in: nova/icehouse
     Assignee: (unassigned) => Rajesh Tailor (rajesh-tailor)

** Changed in: nova/juno
     Assignee: (unassigned) => Rajesh Tailor (rajesh-tailor)

** Changed in: nova/icehouse
    Milestone: None => 2014.1.4

** Tags removed: in-stable-juno

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1361180

Title:
  nova service disable/enable returns 500 on cell environment

Status in OpenStack Compute (Nova):
  Fix Released
Status in OpenStack Compute (nova) icehouse series:
  In Progress
Status in OpenStack Compute (nova) juno series:
  Fix Released

Bug description:
  nova service disable/enable returns 500 on cell environment. Actual
  enable/disable looks processed correctly.

  It also throws following error in nova-api service:
  ValueError: invalid literal for int() with base 10: 'region!child@5'

  How to reproduce:

  $ nova --os-username admin service-list

  Output:
  +----------------+------------------+---------------------+----------+---------+-------+----------------------------+-----------------+
  | Id             | Binary           | Host                | Zone     | Status  | State | Updated_at                 | Disabled Reason |
  +----------------+------------------+---------------------+----------+---------+-------+----------------------------+-----------------+
  | region!child@1 | nova-conductor   | region!child@ubuntu | internal | enabled | up    | 2014-08-18T06:17:36.000000 | -               |
  | region!child@3 | nova-cells       | region!child@ubuntu | internal | enabled | up    | 2014-08-18T06:17:29.000000 | -               |
  | region!child@4 | nova-scheduler   | region!child@ubuntu | internal | enabled | up    | 2014-08-18T06:17:30.000000 | -               |
  | region!child@5 | nova-compute     | region!child@ubuntu | nova     | enabled | up    | 2014-08-18T06:17:31.000000 | -               |
  | region@1       | nova-cells       | region@ubuntu       | internal | enabled | up    | 2014-08-18T06:17:29.000000 | -               |
  | region@2       | nova-cert        | region@ubuntu       | internal | enabled | down  | 2014-08-18T06:08:28.000000 | -               |
  | region@3       | nova-consoleauth | region@ubuntu       | internal | enabled | up    | 2014-08-18T06:17:37.000000 | -               |
  +----------------+------------------+---------------------+----------+---------+-------+----------------------------+-----------------+

  $ nova --os-username admin service-disable 'region!child@ubuntu' nova-
  compute

  The above command gives the following error:
  ERROR (ClientException): Unknown Error (HTTP 500)

  $ nova --os-username admin service-list

  Output:
  +----------------+------------------+---------------------+----------+----------+-------+----------------------------+-----------------+
  | Id             | Binary           | Host                | Zone     | Status   | State | Updated_at                 | Disabled Reason |
  +----------------+------------------+---------------------+----------+----------+-------+----------------------------+-----------------+
  | region!child@1 | nova-conductor   | region!child@ubuntu | internal | enabled  | up    | 2014-08-18T06:19:06.000000 | -               |
  | region!child@3 | nova-cells       | region!child@ubuntu | internal | enabled  | up    | 2014-08-18T06:19:09.000000 | -               |
  | region!child@4 | nova-scheduler   | region!child@ubuntu | internal | enabled  | up    | 2014-08-18T06:19:10.000000 | -               |
  | region!child@5 | nova-compute     | region!child@ubuntu | nova     | disabled | up    | 2014-08-18T06:19:11.000000 | -               |
  | region@1       | nova-cells       | region@ubuntu       | internal | enabled  | up    | 2014-08-18T06:19:09.000000 | -               |
  | region@2       | nova-cert        | region@ubuntu       | internal | enabled  | down  | 2014-08-18T06:08:28.000000 | -               |
  | region@3       | nova-consoleauth | region@ubuntu       | internal | enabled  | up    | 2014-08-18T06:19:07.000000 | -               |
  +----------------+------------------+---------------------+----------+----------+-------+----------------------------+-----------------+

  $ nova --os-username admin service-enable 'region!child@ubuntu' nova-compute
  The above command gives following error:
  ERROR (ClientException): Unknown Error (HTTP 500)

  $ nova --os-username admin service-list
  +----------------+------------------+---------------------+----------+---------+-------+----------------------------+-----------------+
  | Id             | Binary           | Host                | Zone     | Status  | State | Updated_at                 | Disabled Reason |
  +----------------+------------------+---------------------+----------+---------+-------+----------------------------+-----------------+
  | region!child@1 | nova-conductor   | region!child@ubuntu | internal | enabled | up    | 2014-08-18T06:52:37.000000 | -               |
  | region!child@3 | nova-cells       | region!child@ubuntu | internal | enabled | up    | 2014-08-18T06:52:40.000000 | -               |
  | region!child@4 | nova-scheduler   | region!child@ubuntu | internal | enabled | up    | 2014-08-18T06:52:41.000000 | -               |
  | region!child@5 | nova-compute     | region!child@ubuntu | nova     | enabled | up    | 2014-08-18T06:52:42.000000 | -               |
  | region@1       | nova-cells       | region@ubuntu       | internal | enabled | up    | 2014-08-18T06:52:40.000000 | -               |
  | region@2       | nova-cert        | region@ubuntu       | internal | enabled | down  | 2014-08-18T06:08:28.000000 | -               |
  | region@3       | nova-consoleauth | region@ubuntu       | internal | enabled | up    | 2014-08-18T06:52:39.000000 | -               |
  +----------------+------------------+---------------------+----------+---------+-------+----------------------------+-----------------+

  
  The nova-api log shows the following error, after we try to disable/enable the child cell service.

  output of nova-api.log:

  2014-08-18 15:18:23.848 DEBUG nova.api.openstack.wsgi [req-c8e2b844-6aa4-4819-b4f4-38bfe30240f0 admin demo] Action: 'update', body: {"binary": "nova-compute", "host": "region!child@ubuntu"} from (pid=11556) _process_stack /opt/stack/nova/nova/api/openstack/wsgi.py:940
  2014-08-18 15:18:23.848 DEBUG nova.api.openstack.wsgi [req-c8e2b844-6aa4-4819-b4f4-38bfe30240f0 admin demo] Calling method '<bound method ServiceController.update of <nova.api.openstack.compute.contrib.services.ServiceController object at 0x7f16bd608510>>' (Content-type='application/json', Accept='application/json') from (pid=11556) _process_stack /opt/stack/nova/nova/api/openstack/wsgi.py:945
  2014-08-18 15:18:23.930 ERROR object [req-c8e2b844-6aa4-4819-b4f4-38bfe30240f0 admin demo] Error setting Service.id
  2014-08-18 15:18:23.930 TRACE object Traceback (most recent call last):
  2014-08-18 15:18:23.930 TRACE object   File "/opt/stack/nova/nova/objects/base.py", line 70, in setter
  2014-08-18 15:18:23.930 TRACE object     field.coerce(self, name, value))
  2014-08-18 15:18:23.930 TRACE object   File "/opt/stack/nova/nova/objects/fields.py", line 166, in coerce
  2014-08-18 15:18:23.930 TRACE object     return self._type.coerce(obj, attr, value)
  2014-08-18 15:18:23.930 TRACE object   File "/opt/stack/nova/nova/objects/fields.py", line 231, in coerce
  2014-08-18 15:18:23.930 TRACE object     return int(value)
  2014-08-18 15:18:23.930 TRACE object ValueError: invalid literal for int() with base 10: 'region!child@5'
  2014-08-18 15:18:23.930 TRACE object
  2014-08-18 15:18:23.934 ERROR nova.api.openstack [req-c8e2b844-6aa4-4819-b4f4-38bfe30240f0 admin demo] Caught error: invalid literal for int() with base 10: 'region!child@5'
  2014-08-18 15:18:23.934 TRACE nova.api.openstack Traceback (most recent call last):
  2014-08-18 15:18:23.934 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/__init__.py", line 125, in __call__
  2014-08-18 15:18:23.934 TRACE nova.api.openstack     return req.get_response(self.application)
  2014-08-18 15:18:23.934 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
  2014-08-18 15:18:23.934 TRACE nova.api.openstack     application, catch_exc_info=False)
  2014-08-18 15:18:23.934 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
  2014-08-18 15:18:23.934 TRACE nova.api.openstack     app_iter = application(self.environ, start_response)
  2014-08-18 15:18:23.934 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2014-08-18 15:18:23.934 TRACE nova.api.openstack     return resp(environ, start_response)
  2014-08-18 15:18:23.934 TRACE nova.api.openstack   File "/opt/stack/python-keystoneclient/keystoneclient/middleware/auth_token.py", line 663, in __call__
  2014-08-18 15:18:23.934 TRACE nova.api.openstack     return self.app(env, start_response)
  2014-08-18 15:18:23.934 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2014-08-18 15:18:23.934 TRACE nova.api.openstack     return resp(environ, start_response)
  2014-08-18 15:18:23.934 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2014-08-18 15:18:23.934 TRACE nova.api.openstack     return resp(environ, start_response)
  2014-08-18 15:18:23.934 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
  2014-08-18 15:18:23.934 TRACE nova.api.openstack     response = self.app(environ, start_response)
  2014-08-18 15:18:23.934 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2014-08-18 15:18:23.934 TRACE nova.api.openstack     return resp(environ, start_response)
  2014-08-18 15:18:23.934 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
  2014-08-18 15:18:23.934 TRACE nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
  2014-08-18 15:18:23.934 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
  2014-08-18 15:18:23.934 TRACE nova.api.openstack     return self.func(req, *args, **kwargs)
  2014-08-18 15:18:23.934 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 917, in __call__
  2014-08-18 15:18:23.934 TRACE nova.api.openstack     content_type, body, accept)
  2014-08-18 15:18:23.934 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 983, in _process_stack
  2014-08-18 15:18:23.934 TRACE nova.api.openstack     action_result = self.dispatch(meth, request, action_args)
  2014-08-18 15:18:23.934 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 1070, in dispatch
  2014-08-18 15:18:23.934 TRACE nova.api.openstack     return method(req=request, **action_args)
  2014-08-18 15:18:23.934 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/compute/contrib/services.py", line 206, in update
  2014-08-18 15:18:23.934 TRACE nova.api.openstack     self.host_api.service_update(context, host, binary, status_detail)
  2014-08-18 15:18:23.934 TRACE nova.api.openstack   File "/opt/stack/nova/nova/compute/cells_api.py", line 579, in service_update
  2014-08-18 15:18:23.934 TRACE nova.api.openstack     db_service)
  2014-08-18 15:18:23.934 TRACE nova.api.openstack   File "/opt/stack/nova/nova/objects/service.py", line 67, in _from_db_object
  2014-08-18 15:18:23.934 TRACE nova.api.openstack     service[key] = db_service[key]
  2014-08-18 15:18:23.934 TRACE nova.api.openstack   File "/opt/stack/nova/nova/objects/base.py", line 398, in __setitem__
  2014-08-18 15:18:23.934 TRACE nova.api.openstack     setattr(self, name, value)
  2014-08-18 15:18:23.934 TRACE nova.api.openstack   File "/opt/stack/nova/nova/objects/base.py", line 70, in setter
  2014-08-18 15:18:23.934 TRACE nova.api.openstack     field.coerce(self, name, value))
  2014-08-18 15:18:23.934 TRACE nova.api.openstack   File "/opt/stack/nova/nova/objects/fields.py", line 166, in coerce
  2014-08-18 15:18:23.934 TRACE nova.api.openstack     return self._type.coerce(obj, attr, value)
  2014-08-18 15:18:23.934 TRACE nova.api.openstack   File "/opt/stack/nova/nova/objects/fields.py", line 231, in coerce
  2014-08-18 15:18:23.934 TRACE nova.api.openstack     return int(value)
  2014-08-18 15:18:23.934 TRACE nova.api.openstack ValueError: invalid literal for int() with base 10: 'region!child@5'

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1361180/+subscriptions
References

[Bug 1361180] [NEW] nova service disable/enable returns 500 on cell environment
From: Rajesh Tailor, 2014-08-25