yahoo-eng-team team mailing list archive

[OpenStack Infra <1706772@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.openstack.org/487925
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c2d33c3271370358d48553233b41bf9119d834fb
Submitter: Jenkins
Branch:    master

commit c2d33c3271370358d48553233b41bf9119d834fb
Author: EdLeafe <ed@xxxxxxxxx>
Date:   Wed Jul 26 23:07:38 2017 +0000

    Handle ironicclient failures in Ironic driver
    
    The ironic driver uses ironicclient to get the list of nodes from the
    Ironic service. If that service is not running, or something otherwise
    prevents the ironicclient's call from succeeding, and unhandled
    exception will occur. This patch adds a generic exception handler for
    this call, logging an error and continuing.
    
    Co-Authored-By: Lucas Alvares Gomes <lucasagomes@xxxxxxxxx>
    
    Closes-Bug: #1706772
    
    Change-Id: I4ebebcf221dfe29e2aa125f5956aec10108f8fbe


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1706772

Title:
  InternalServerError: Internal Server Error (HTTP 500) in n-cpu logs on
  startup with Ironic driver

Status in Ironic:
  Confirmed
Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) ocata series:
  In Progress

Bug description:
  The init_host stacktraces on startup, killing the thread:

  http://logs.openstack.org/49/484949/14/check/gate-tempest-dsvm-ironic-
  ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial-
  nv/69b18d7/logs/screen-n-cpu.txt.gz?level=TRACE

  Jul 24 22:26:26.423871 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service [None req-414af81a-44e7-4a67-9cbd-74e549deb41b None None] Error starting thread.: InternalServerError: Internal Server Error (HTTP 500)
  Jul 24 22:26:26.423993 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service Traceback (most recent call last):
  Jul 24 22:26:26.424084 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service   File "/usr/local/lib/python2.7/dist-packages/oslo_service/service.py", line 721, in run_service
  Jul 24 22:26:26.424167 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service     service.start()
  Jul 24 22:26:26.424250 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service   File "/opt/stack/new/nova/nova/service.py", line 143, in start
  Jul 24 22:26:26.424330 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service     self.manager.init_host()
  Jul 24 22:26:26.424417 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service   File "/opt/stack/new/nova/nova/compute/manager.py", line 1100, in init_host
  Jul 24 22:26:26.424498 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service     self.driver.init_host(host=self.host)
  Jul 24 22:26:26.424638 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service   File "/opt/stack/new/nova/nova/virt/ironic/driver.py", line 449, in init_host
  Jul 24 22:26:26.424717 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service     self._refresh_cache()
  Jul 24 22:26:26.424798 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service   File "/opt/stack/new/nova/nova/virt/ironic/driver.py", line 608, in _refresh_cache
  Jul 24 22:26:26.425096 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service     for node in self._get_node_list(detail=True, limit=0):
  Jul 24 22:26:26.425187 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service   File "/opt/stack/new/nova/nova/virt/ironic/driver.py", line 527, in _get_node_list
  Jul 24 22:26:26.425278 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service     node_list = self.ironicclient.call("node.list", **kwargs)
  Jul 24 22:26:26.425359 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service   File "/opt/stack/new/nova/nova/virt/ironic/client_wrapper.py", line 146, in call
  Jul 24 22:26:26.425440 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service     return self._multi_getattr(client, method)(*args, **kwargs)
  Jul 24 22:26:26.425536 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service   File "/usr/local/lib/python2.7/dist-packages/ironicclient/v1/node.py", line 143, in list
  Jul 24 22:26:26.425614 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service     limit=limit)
  Jul 24 22:26:26.425708 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service   File "/usr/local/lib/python2.7/dist-packages/ironicclient/common/base.py", line 149, in _list_pagination
  Jul 24 22:26:26.425788 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service     resp, body = self.api.json_request('GET', url)
  Jul 24 22:26:26.425866 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service   File "/usr/local/lib/python2.7/dist-packages/ironicclient/common/http.py", line 558, in json_request
  Jul 24 22:26:26.425945 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service     resp = self._http_request(url, method, **kwargs)
  Jul 24 22:26:26.426027 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service   File "/usr/local/lib/python2.7/dist-packages/ironicclient/common/http.py", line 188, in wrapper
  Jul 24 22:26:26.426114 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service     return func(self, url, method, **kwargs)
  Jul 24 22:26:26.426198 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service   File "/usr/local/lib/python2.7/dist-packages/ironicclient/common/http.py", line 540, in _http_request
  Jul 24 22:26:26.426277 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service     error_json.get('debuginfo'), method, url)
  Jul 24 22:26:26.426355 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service InternalServerError: Internal Server Error (HTTP 500)
  Jul 24 22:26:26.426437 ubuntu-xenial-osic-cloud1-disk-10052658 nova-compute[13755]: ERROR oslo_service.service

  I guess the Ironic API service isn't yet up at that point?

To manage notifications about this bug go to:
https://bugs.launchpad.net/ironic/+bug/1706772/+subscriptions