← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #66147

[Bug 1706772] [NEW] InternalServerError: Internal Server Error (HTTP 500) in n-cpu logs on startup with Ironic driver

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Since this change in Pike:

https://review.openstack.org/#/c/484949/

The init_host stacktraces on startup, killing the thread:

http://logs.openstack.org/49/484949/14/check/gate-tempest-dsvm-ironic-
ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial-
nv/69b18d7/logs/screen-n-cpu.txt.gz?level=TRACE

Jul 24 22:26:26.423871 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service [None req-414af81a-44e7-4a67-9cbd-74e549deb41b None None] Error starting thread.: InternalServerError: Internal Server Error (HTTP 500)
Jul 24 22:26:26.423993 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service Traceback (most recent call last):
Jul 24 22:26:26.424084 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service   File "/usr/local/lib/python2.7/dist-packages/oslo_service/service.py", line 721, in run_service
Jul 24 22:26:26.424167 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service     service.start()
Jul 24 22:26:26.424250 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service   File "/opt/stack/new/nova/nova/service.py", line 143, in start
Jul 24 22:26:26.424330 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service     self.manager.init_host()
Jul 24 22:26:26.424417 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service   File "/opt/stack/new/nova/nova/compute/manager.py", line 1100, in init_host
Jul 24 22:26:26.424498 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service     self.driver.init_host(host=self.host)
Jul 24 22:26:26.424638 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service   File "/opt/stack/new/nova/nova/virt/ironic/driver.py", line 449, in init_host
Jul 24 22:26:26.424717 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service     self._refresh_cache()
Jul 24 22:26:26.424798 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service   File "/opt/stack/new/nova/nova/virt/ironic/driver.py", line 608, in _refresh_cache
Jul 24 22:26:26.425096 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service     for node in self._get_node_list(detail=True, limit=0):
Jul 24 22:26:26.425187 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service   File "/opt/stack/new/nova/nova/virt/ironic/driver.py", line 527, in _get_node_list
Jul 24 22:26:26.425278 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service     node_list = self.ironicclient.call("node.list", **kwargs)
Jul 24 22:26:26.425359 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service   File "/opt/stack/new/nova/nova/virt/ironic/client_wrapper.py", line 146, in call
Jul 24 22:26:26.425440 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service     return self._multi_getattr(client, method)(*args, **kwargs)
Jul 24 22:26:26.425536 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service   File "/usr/local/lib/python2.7/dist-packages/ironicclient/v1/node.py", line 143, in list
Jul 24 22:26:26.425614 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service     limit=limit)
Jul 24 22:26:26.425708 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service   File "/usr/local/lib/python2.7/dist-packages/ironicclient/common/base.py", line 149, in _list_pagination
Jul 24 22:26:26.425788 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service     resp, body = self.api.json_request('GET', url)
Jul 24 22:26:26.425866 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service   File "/usr/local/lib/python2.7/dist-packages/ironicclient/common/http.py", line 558, in json_request
Jul 24 22:26:26.425945 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service     resp = self._http_request(url, method, **kwargs)
Jul 24 22:26:26.426027 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service   File "/usr/local/lib/python2.7/dist-packages/ironicclient/common/http.py", line 188, in wrapper
Jul 24 22:26:26.426114 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service     return func(self, url, method, **kwargs)
Jul 24 22:26:26.426198 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service   File "/usr/local/lib/python2.7/dist-packages/ironicclient/common/http.py", line 540, in _http_request
Jul 24 22:26:26.426277 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service     error_json.get('debuginfo'), method, url)
Jul 24 22:26:26.426355 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service InternalServerError: Internal Server Error (HTTP 500)
Jul 24 22:26:26.426437 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service 

I guess the Ironic API service isn't yet up at that point?

** Affects: nova
     Importance: High
         Status: Confirmed


** Tags: compute ironic pike-rc-potential

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1706772

Title:
  InternalServerError: Internal Server Error (HTTP 500) in n-cpu logs on
  startup with Ironic driver

Status in OpenStack Compute (nova):
  Confirmed

Bug description:
  Since this change in Pike:

  https://review.openstack.org/#/c/484949/

  The init_host stacktraces on startup, killing the thread:

  http://logs.openstack.org/49/484949/14/check/gate-tempest-dsvm-ironic-
  ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial-
  nv/69b18d7/logs/screen-n-cpu.txt.gz?level=TRACE

  Jul 24 22:26:26.423871 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service [None req-414af81a-44e7-4a67-9cbd-74e549deb41b None None] Error starting thread.: InternalServerError: Internal Server Error (HTTP 500)
  Jul 24 22:26:26.423993 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service Traceback (most recent call last):
  Jul 24 22:26:26.424084 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service   File "/usr/local/lib/python2.7/dist-packages/oslo_service/service.py", line 721, in run_service
  Jul 24 22:26:26.424167 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service     service.start()
  Jul 24 22:26:26.424250 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service   File "/opt/stack/new/nova/nova/service.py", line 143, in start
  Jul 24 22:26:26.424330 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service     self.manager.init_host()
  Jul 24 22:26:26.424417 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service   File "/opt/stack/new/nova/nova/compute/manager.py", line 1100, in init_host
  Jul 24 22:26:26.424498 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service     self.driver.init_host(host=self.host)
  Jul 24 22:26:26.424638 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service   File "/opt/stack/new/nova/nova/virt/ironic/driver.py", line 449, in init_host
  Jul 24 22:26:26.424717 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service     self._refresh_cache()
  Jul 24 22:26:26.424798 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service   File "/opt/stack/new/nova/nova/virt/ironic/driver.py", line 608, in _refresh_cache
  Jul 24 22:26:26.425096 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service     for node in self._get_node_list(detail=True, limit=0):
  Jul 24 22:26:26.425187 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service   File "/opt/stack/new/nova/nova/virt/ironic/driver.py", line 527, in _get_node_list
  Jul 24 22:26:26.425278 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service     node_list = self.ironicclient.call("node.list", **kwargs)
  Jul 24 22:26:26.425359 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service   File "/opt/stack/new/nova/nova/virt/ironic/client_wrapper.py", line 146, in call
  Jul 24 22:26:26.425440 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service     return self._multi_getattr(client, method)(*args, **kwargs)
  Jul 24 22:26:26.425536 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service   File "/usr/local/lib/python2.7/dist-packages/ironicclient/v1/node.py", line 143, in list
  Jul 24 22:26:26.425614 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service     limit=limit)
  Jul 24 22:26:26.425708 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service   File "/usr/local/lib/python2.7/dist-packages/ironicclient/common/base.py", line 149, in _list_pagination
  Jul 24 22:26:26.425788 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service     resp, body = self.api.json_request('GET', url)
  Jul 24 22:26:26.425866 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service   File "/usr/local/lib/python2.7/dist-packages/ironicclient/common/http.py", line 558, in json_request
  Jul 24 22:26:26.425945 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service     resp = self._http_request(url, method, **kwargs)
  Jul 24 22:26:26.426027 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service   File "/usr/local/lib/python2.7/dist-packages/ironicclient/common/http.py", line 188, in wrapper
  Jul 24 22:26:26.426114 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service     return func(self, url, method, **kwargs)
  Jul 24 22:26:26.426198 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service   File "/usr/local/lib/python2.7/dist-packages/ironicclient/common/http.py", line 540, in _http_request
  Jul 24 22:26:26.426277 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service     error_json.get('debuginfo'), method, url)
  Jul 24 22:26:26.426355 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service InternalServerError: Internal Server Error (HTTP 500)
  Jul 24 22:26:26.426437 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service 

  I guess the Ironic API service isn't yet up at that point?

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1706772/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next