yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1652627] Re: nova-compute died immediatelly after start

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Launchpad Bug Tracker <1652627@xxxxxxxxxxxxxxxxxx>
Date: Tue, 07 Mar 2017 04:17:37 -0000
Reply-to: Bug 1652627 <1652627@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
[Expired for OpenStack Compute (nova) because there has been no activity
for 60 days.]

** Changed in: nova
       Status: Incomplete => Expired

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1652627

Title:
  nova-compute died immediatelly after start

Status in OpenStack Compute (nova):
  Expired

Bug description:
  After complete install openstack newton of nova-compute, when I start
  the nova-compute service, it dies immediately with error message:

  2016-12-26 20:49:36.844 27086 ERROR oslo_service.service [req-44cf0677-f4c0-4fa1-9649-48e63c22486a - - - - -] Error starting thread.
  2016-12-26 20:49:36.844 27086 ERROR oslo_service.service Traceback (most recent call last):
  2016-12-26 20:49:36.844 27086 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/oslo_service/service.py", line 708, in run_service
  2016-12-26 20:49:36.844 27086 ERROR oslo_service.service     service.start()
  2016-12-26 20:49:36.844 27086 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/service.py", line 117, in start
  2016-12-26 20:49:36.844 27086 ERROR oslo_service.service     self.manager.init_host()
  2016-12-26 20:49:36.844 27086 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1139, in init_host
  2016-12-26 20:49:36.844 27086 ERROR oslo_service.service     context, self.host, expected_attrs=['info_cache', 'metadata'])
  2016-12-26 20:49:36.844 27086 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 177, in wrapper
  2016-12-26 20:49:36.844 27086 ERROR oslo_service.service     args, kwargs)
  2016-12-26 20:49:36.844 27086 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/conductor/rpcapi.py", line 236, in object_class_action_versions
  2016-12-26 20:49:36.844 27086 ERROR oslo_service.service     args=args, kwargs=kwargs)
  2016-12-26 20:49:36.844 27086 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 169, in call
  2016-12-26 20:49:36.844 27086 ERROR oslo_service.service     retry=self.retry)
  2016-12-26 20:49:36.844 27086 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 97, in _send
  2016-12-26 20:49:36.844 27086 ERROR oslo_service.service     timeout=timeout, retry=retry)
  2016-12-26 20:49:36.844 27086 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 464, in send
  2016-12-26 20:49:36.844 27086 ERROR oslo_service.service     retry=retry)
  2016-12-26 20:49:36.844 27086 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 455, in _send
  2016-12-26 20:49:36.844 27086 ERROR oslo_service.service     raise result
  2016-12-26 20:49:36.844 27086 ERROR oslo_service.service RemoteError: Remote error: UnsupportedVersion Endpoint does not support RPC version 3.0
  2016-12-26 20:49:36.844 27086 ERROR oslo_service.service [u'Traceback (most recent call last):\n', u'  File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply\n    incoming.message))\n', u'  File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 185, in _dispatch\n    raise UnsupportedVersion(version)\n', u'UnsupportedVersion: Endpoint does not support RPC version 3.0\n'].
  2016-12-26 20:49:36.844 27086 ERROR oslo_service.service

  I have the same nova version on controller node and compute node, and
  also when I try to run "openstack compute service list" on the
  controller node, it has error:

  Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <type 'exceptions.ValueError'> (HTTP 500) (Request-ID: req-499359d4-4362-48dc-8d7f-260d29c6900f)

  complete log:

  START with options: [u'--debug', u'compute', u'service', u'list']
  options: Namespace(access_key='', access_secret='***', access_token='***', access_token_endpoint='', access_token_type='', auth_type='', auth_url='http://controller1:35357/v3', authorization_code='', cacert=None, cert='', client_id='', client_secret='***', cloud='', consumer_key='', consumer_secret='***', debug=True, default_domain='default', default_domain_id='', default_domain_name='', deferred_help=False, discovery_endpoint='', domain_id='', domain_name='', endpoint='', identity_provider='', identity_provider_url='', insecure=None, interface='', key='', log_file=None, old_profile=None, openid_scope='', os_beta_command=False, os_compute_api_version='', os_identity_api_version='3', os_image_api_version='2', os_network_api_version='', os_object_api_version='', os_project_id=None, os_project_name=None, os_volume_api_version='', passcode='', password='***', profile=None, project_domain_id='', project_domain_name='default', project_id='', project_name='admin', protocol='', redirect_uri='', region_name='', timing=False, token='***', trust_id='', url='', user_domain_id='', user_domain_name='default', user_id='', username='admin', verbose_level=3, verify=None)
  Auth plugin password selected
  auth_config_hook(): {'auth_type': 'password', 'beta_command': False, u'compute_api_version': u'2', 'key': None, u'database_api_version': u'1.0', u'interface': None, 'auth_url': 'http://controller1:35357/v3', u'network_api_version': u'2', u'image_format': u'qcow2', 'networks': [], u'image_api_version': '2', 'verify': True, u'dns_api_version': u'2', u'object_store_api_version': u'1', 'username': 'admin', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': u'1', 'auth': {'user_domain_name': 'default', 'project_name': 'admin', 'project_domain_name': 'default'}, 'default_domain': 'default', 'debug': True, u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': u'1', 'timing': False, 'password': 'D34EC9030D', 'cacert': None, u'key_manager_api_version': u'v1', u'metering_api_version': u'2', 'deferred_help': False, u'identity_api_version': '3', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'container_api_version': u'1', u'disable_vendor_agent': {}}
  defaults: {u'auth_type': 'password', u'compute_api_version': u'2', 'key': None, u'database_api_version': u'1.0', 'api_timeout': None, u'baremetal_api_version': u'1', u'image_api_version': u'2', 'cacert': None, u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': u'1', u'interface': None, u'network_api_version': u'2', u'image_format': u'qcow2', u'key_manager_api_version': u'v1', u'metering_api_version': u'2', 'verify': True, u'identity_api_version': u'2.0', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'container_api_version': u'1', u'dns_api_version': u'2', u'object_store_api_version': u'1', u'disable_vendor_agent': {}}
  cloud cfg: {'auth_type': 'password', 'beta_command': False, u'compute_api_version': u'2', u'orchestration_api_version': u'1', u'database_api_version': u'1.0', 'timing': False, 'auth_url': 'http://controller1:35357/v3', u'network_api_version': u'2', u'image_format': u'qcow2', 'networks': [], u'image_api_version': '2', 'verify': True, u'dns_api_version': u'2', u'object_store_api_version': u'1', 'username': 'admin', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': u'1', 'auth': {'username': 'admin', 'project_name': 'admin', 'user_domain_name': 'default', 'auth_url': 'http://controller1:35357/v3', 'password': '***', 'project_domain_name': 'default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', 'key': None, u'interface': None, 'password': '***', 'cacert': None, u'key_manager_api_version': u'v1', u'metering_api_version': u'2', 'deferred_help': False, u'identity_api_version': '3', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', 'debug': True, u'disable_vendor_agent': {}}
  compute API version 2, cmd group openstack.compute.v2
  network API version 2, cmd group openstack.network.v2
  image API version 2, cmd group openstack.image.v2
  volume API version 2, cmd group openstack.volume.v2
  identity API version 3, cmd group openstack.identity.v3
  object_store API version 1, cmd group openstack.object_store.v1
  neutronclient API version 2, cmd group openstack.neutronclient.v2
  Auth plugin password selected
  auth_config_hook(): {'auth_type': 'password', 'beta_command': False, u'compute_api_version': u'2', 'key': None, u'database_api_version': u'1.0', u'interface': None, 'auth_url': 'http://controller1:35357/v3', u'network_api_version': u'2', u'image_format': u'qcow2', 'networks': [], u'image_api_version': '2', 'verify': True, u'dns_api_version': u'2', u'object_store_api_version': u'1', 'username': 'admin', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': u'1', 'auth': {'user_domain_name': 'default', 'project_name': 'admin', 'project_domain_name': 'default'}, 'default_domain': 'default', 'debug': True, u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': u'1', 'timing': False, 'password': 'D34EC9030D', 'cacert': None, u'key_manager_api_version': u'v1', u'metering_api_version': u'2', 'deferred_help': False, u'identity_api_version': '3', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'container_api_version': u'1', u'disable_vendor_agent': {}}
  command: compute service list -> openstackclient.compute.v2.service.ListService
  Using auth plugin: password
  Using parameters {'username': 'admin', 'project_name': 'admin', 'user_domain_name': 'default', 'auth_url': 'http://controller1:35357/v3', 'password': '***', 'project_domain_name': 'default'}
  Get auth_ref
  REQ: curl -g -i -X GET http://controller1:35357/v3 -H "Accept: application/json" -H "User-Agent: osc-lib keystoneauth1/2.12.1 python-requests/2.10.0 CPython/2.7.12"
  Starting new HTTP connection (1): controller1
  "GET /v3 HTTP/1.1" 200 251
  RESP: [200] Date: Mon, 26 Dec 2016 14:19:32 GMT Server: Apache/2.4.18 (Ubuntu) Vary: X-Auth-Token X-Distribution: Ubuntu x-openstack-request-id: req-ce039729-7ced-4ee2-9a0c-037232cc5668 Content-Length: 251 Keep-Alive: timeout=5, max=100 Connection: Keep-Alive Content-Type: application/json
  RESP BODY: {"version": {"status": "stable", "updated": "2016-10-06T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.7", "links": [{"href": "http://controller1:35357/v3/";, "rel": "self"}]}}

  Making authentication request to http://controller1:35357/v3/auth/tokens
  "POST /v3/auth/tokens HTTP/1.1" 201 2276
  {"token": {"is_domain": false, "methods": ["password"], "roles": [{"id": "8305e967804a4a39ad21584ae093e972", "name": "admin"}], "expires_at": "2016-12-26T15:19:32.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "057783c4e1334f0db55b6187bfcefab2", "name": "admin"}, "catalog": [{"endpoints": [{"url": "http://controller1:8774/v2.1/057783c4e1334f0db55b6187bfcefab2";, "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "031c2466602d42a9a4f00d1efdd2395d"}, {"url": "http://controller1:8774/v2.1/057783c4e1334f0db55b6187bfcefab2";, "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "297a33a473a94e9fb776d628e1fe4012"}, {"url": "http://controller1:8774/v2.1/057783c4e1334f0db55b6187bfcefab2";, "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "2e2c2107bd654df4855b952d74399848"}], "type": "compute", "id": "bc67c41b3653459e97bc82cfc203e294", "name": "nova"}, {"endpoints": [{"url": "http://controller1:35357/v3/";, "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "51963649520e4926872871ab79931518"}, {"url": "http://controller1:35357/v3/";, "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "630980effb51495db6e4c1b6a990acf7"}, {"url": "http://controller1:5000/v3/";, "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "d1e4503b71554d89b6a1f0c67d1e7f6f"}], "type": "identity", "id": "f62a4310574d4037a4e08cf4189dd2a2", "name": "keystone"}, {"endpoints": [{"url": "http://controller1:9292";, "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "8544172e19c747488ede2da28edc3ebc"}, {"url": "http://controller1:9292";, "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "bad7c412ed404c5d86a86cd3ea82b22f"}, {"url": "http://controller1:9292";, "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "c4f63f643f634e26ac8ec508c57de11a"}], "type": "image", "id": "f64d65ba68244068bf64adefd14d9beb", "name": "glance"}], "user": {"domain": {"id": "default", "name": "Default"}, "id": "6c9e552cae4c42aa81e2c0eed68bdb0d", "name": "admin"}, "audit_ids": ["vs0KOX9qQg258Nc5pTf0vQ"], "issued_at": "2016-12-26T14:19:32.000000Z"}}
  run(Namespace(columns=[], formatter='table', host=None, long=False, max_width=0, noindent=False, quote_mode='nonnumeric', service=None))
  Instantiating compute client for API Version Major: 2, Minor: 0
  Making authentication request to http://controller1:35357/v3/auth/tokens
  "POST /v3/auth/tokens HTTP/1.1" 201 2276
  {"token": {"is_domain": false, "methods": ["password"], "roles": [{"id": "8305e967804a4a39ad21584ae093e972", "name": "admin"}], "expires_at": "2016-12-26T15:19:32.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "057783c4e1334f0db55b6187bfcefab2", "name": "admin"}, "catalog": [{"endpoints": [{"url": "http://controller1:8774/v2.1/057783c4e1334f0db55b6187bfcefab2";, "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "031c2466602d42a9a4f00d1efdd2395d"}, {"url": "http://controller1:8774/v2.1/057783c4e1334f0db55b6187bfcefab2";, "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "297a33a473a94e9fb776d628e1fe4012"}, {"url": "http://controller1:8774/v2.1/057783c4e1334f0db55b6187bfcefab2";, "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "2e2c2107bd654df4855b952d74399848"}], "type": "compute", "id": "bc67c41b3653459e97bc82cfc203e294", "name": "nova"}, {"endpoints": [{"url": "http://controller1:35357/v3/";, "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "51963649520e4926872871ab79931518"}, {"url": "http://controller1:35357/v3/";, "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "630980effb51495db6e4c1b6a990acf7"}, {"url": "http://controller1:5000/v3/";, "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "d1e4503b71554d89b6a1f0c67d1e7f6f"}], "type": "identity", "id": "f62a4310574d4037a4e08cf4189dd2a2", "name": "keystone"}, {"endpoints": [{"url": "http://controller1:9292";, "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "8544172e19c747488ede2da28edc3ebc"}, {"url": "http://controller1:9292";, "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "bad7c412ed404c5d86a86cd3ea82b22f"}, {"url": "http://controller1:9292";, "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "c4f63f643f634e26ac8ec508c57de11a"}], "type": "image", "id": "f64d65ba68244068bf64adefd14d9beb", "name": "glance"}], "user": {"domain": {"id": "default", "name": "Default"}, "id": "6c9e552cae4c42aa81e2c0eed68bdb0d", "name": "admin"}, "audit_ids": ["kcbQsErPSd-Op13ULtXL6g"], "issued_at": "2016-12-26T14:19:32.000000Z"}}
  REQ: curl -g -i -X GET http://controller1:8774/v2.1/057783c4e1334f0db55b6187bfcefab2/os-services -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}a3778de62c6816c3939997781e0be41e3b7b339d"
  Starting new HTTP connection (1): controller1
  "GET /v2.1/057783c4e1334f0db55b6187bfcefab2/os-services HTTP/1.1" 500 194
  RESP: [500] Openstack-Api-Version: compute 2.1 X-Openstack-Nova-Api-Version: 2.1 Vary: OpenStack-API-Version, X-OpenStack-Nova-API-Version Content-Type: application/json; charset=UTF-8 Content-Length: 194 X-Compute-Request-Id: req-e9d97dd6-a861-414c-a979-0fb40ed651b9 Date: Mon, 26 Dec 2016 14:19:32 GMT Connection: keep-alive
  RESP BODY: {"computeFault": {"message": "Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.\n<type 'exceptions.ValueError'>", "code": 500}}

  GET call to compute for http://controller1:8774/v2.1/057783c4e1334f0db55b6187bfcefab2/os-services used request id req-e9d97dd6-a861-414c-a979-0fb40ed651b9
  Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <type 'exceptions.ValueError'> (HTTP 500) (Request-ID: req-e9d97dd6-a861-414c-a979-0fb40ed651b9)
  Traceback (most recent call last):
    File "/usr/lib/python2.7/dist-packages/cliff/app.py", line 387, in run_subcommand
      result = cmd.run(parsed_args)
    File "/usr/lib/python2.7/dist-packages/osc_lib/command/command.py", line 41, in run
      return super(Command, self).run(parsed_args)
    File "/usr/lib/python2.7/dist-packages/cliff/display.py", line 100, in run
      column_names, data = self.take_action(parsed_args)
    File "/usr/lib/python2.7/dist-packages/openstackclient/compute/v2/service.py", line 110, in take_action
      parsed_args.service)
    File "/usr/lib/python2.7/dist-packages/novaclient/v2/services.py", line 52, in list
      return self._list(url, "services")
    File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 249, in _list
      resp, body = self.api.client.get(url)
    File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 187, in get
      return self.request(url, 'GET', **kwargs)
    File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 117, in request
      raise exceptions.from_response(resp, body, url, method)
  ClientException: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <type 'exceptions.ValueError'> (HTTP 500) (Request-ID: req-e9d97dd6-a861-414c-a979-0fb40ed651b9)
  clean_up ListService: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <type 'exceptions.ValueError'> (HTTP 500) (Request-ID: req-e9d97dd6-a861-414c-a979-0fb40ed651b9)
  Traceback (most recent call last):
    File "/usr/lib/python2.7/dist-packages/osc_lib/shell.py", line 135, in run
      ret_val = super(OpenStackShell, self).run(argv)
    File "/usr/lib/python2.7/dist-packages/cliff/app.py", line 267, in run
      result = self.run_subcommand(remainder)
    File "/usr/lib/python2.7/dist-packages/osc_lib/shell.py", line 180, in run_subcommand
      ret_value = super(OpenStackShell, self).run_subcommand(argv)
    File "/usr/lib/python2.7/dist-packages/cliff/app.py", line 387, in run_subcommand
      result = cmd.run(parsed_args)
    File "/usr/lib/python2.7/dist-packages/osc_lib/command/command.py", line 41, in run
      return super(Command, self).run(parsed_args)
    File "/usr/lib/python2.7/dist-packages/cliff/display.py", line 100, in run
      column_names, data = self.take_action(parsed_args)
    File "/usr/lib/python2.7/dist-packages/openstackclient/compute/v2/service.py", line 110, in take_action
      parsed_args.service)
    File "/usr/lib/python2.7/dist-packages/novaclient/v2/services.py", line 52, in list
      return self._list(url, "services")
    File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 249, in _list
      resp, body = self.api.client.get(url)
    File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 187, in get
      return self.request(url, 'GET', **kwargs)
    File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 117, in request
      raise exceptions.from_response(resp, body, url, method)
  ClientException: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <type 'exceptions.ValueError'> (HTTP 500) (Request-ID: req-e9d97dd6-a861-414c-a979-0fb40ed651b9)

  END return value: 1

  
  nova-api, nova-consoleauth, nova-scheduler, nova-conductor and nova-novncproxy are started succesfully on the controller node.

  root@maas:~# dpkg -l | grep nova
  ii  nova-api                           2:14.0.1-0ubuntu1~cloud0              all          OpenStack Compute - API frontend
  ii  nova-common                        2:14.0.1-0ubuntu1~cloud0              all          OpenStack Compute - common files
  ii  nova-conductor                     2:14.0.1-0ubuntu1~cloud0              all          OpenStack Compute - conductor service
  ii  nova-consoleauth                   2:14.0.1-0ubuntu1~cloud0              all          OpenStack Compute - Console Authenticator
  ii  nova-novncproxy                    2:14.0.1-0ubuntu1~cloud0              all          OpenStack Compute - NoVNC proxy
  ii  nova-scheduler                     2:14.0.1-0ubuntu1~cloud0              all          OpenStack Compute - virtual machine scheduler
  ii  python-nova                        2:14.0.1-0ubuntu1~cloud0              all          OpenStack Compute Python libraries
  ii  python-novaclient                  2:6.0.0-0ubuntu1~cloud0               all          client library for OpenStack Compute API - Python 2.7

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1652627/+subscriptions
References

[Bug 1652627] [NEW] nova-compute died immediatelly after start
From: ariee, 2016-12-26