yahoo-eng-team team mailing list archive

[Matt Riedemann <mriedem@xxxxxxxxxx>]

This is superseded by https://review.openstack.org/#/c/211097/

** Changed in: nova
       Status: In Progress => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1404331

Title:
  ironic driver logs incorrect error message when node in unexpected
  state

Status in OpenStack Compute (nova):
  Invalid

Bug description:
  When an Ironic node is not in the expected state (eg, it somehow is
  out of sync with the nova driver), an incorrect error message is
  logged in Nova.

  This showed up while testing changes to Ironic's state machine (so the
  node being in the wrong state is not Nova's fault; I broke something
  in Ironic to cause that). Regardless of the cause of the InvalidState
  error, our Nova driver should handle it better.

  Here is a copy of the trace from this test run:
  http://logs.openstack.org/83/140883/6/check/check-tempest-dsvm-ironic-pxe_ssh/369aebc/logs/screen-n-cpu.txt.gz?#_2014-12-19_16_52_57_030

  
  2014-12-19 16:52:57.030 WARNING ironicclient.common.http [req-7059788d-3695-4b22-851a-bec30922e823 demo demo] Request returned failure status.
  2014-12-19 16:52:57.030 WARNING nova.virt.ironic.client_wrapper [req-7059788d-3695-4b22-851a-bec30922e823 demo demo] Error contacting Ironic server for 'node.update'. Attempt 59 of 60
  ...
  {"error_message": "{\"debuginfo\": null, \"faultcode\": \"Client\", \"faultstring\": \"Node 07a3ce7c-0726-4fc2-a94b-a707d0450b5a can not be updated while a state transition is in progress.\"}"}
   log_http_response /usr/local/lib/python2.7/dist-packages/ironicclient/common/http.py:119
  2014-12-19 16:52:59.196 WARNING ironicclient.common.http [req-7059788d-3695-4b22-851a-bec30922e823 demo demo] Request returned failure status.
  2014-12-19 16:52:59.196 ERROR nova.virt.ironic.client_wrapper [req-7059788d-3695-4b22-851a-bec30922e823 demo demo] Error contacting Ironic server for 'node.update'. Attempt 60 of 60
  2014-12-19 16:52:59.197 ERROR nova.compute.manager [req-7059788d-3695-4b22-851a-bec30922e823 demo demo] [instance: 604b621c-2103-4343-85f4-acaef2b0eb18] Setting instance vm_state to ERROR
  2014-12-19 16:52:59.197 31679 TRACE nova.compute.manager [instance: 604b621c-2103-4343-85f4-acaef2b0eb18] Traceback (most recent call last):
  2014-12-19 16:52:59.197 31679 TRACE nova.compute.manager [instance: 604b621c-2103-4343-85f4-acaef2b0eb18]   File "/opt/stack/new/nova/nova/compute/manager.py", line 6148, in _error_out_instance_on_exception
  2014-12-19 16:52:59.197 31679 TRACE nova.compute.manager [instance: 604b621c-2103-4343-85f4-acaef2b0eb18]     yield
  2014-12-19 16:52:59.197 31679 TRACE nova.compute.manager [instance: 604b621c-2103-4343-85f4-acaef2b0eb18]   File "/opt/stack/new/nova/nova/compute/manager.py", line 2865, in rebuild_instance
  2014-12-19 16:52:59.197 31679 TRACE nova.compute.manager [instance: 604b621c-2103-4343-85f4-acaef2b0eb18]     self.driver.rebuild(**kwargs)
  2014-12-19 16:52:59.197 31679 TRACE nova.compute.manager [instance: 604b621c-2103-4343-85f4-acaef2b0eb18]   File "/opt/stack/new/nova/nova/virt/ironic/driver.py", line 1007, in rebuild
  2014-12-19 16:52:59.197 31679 TRACE nova.compute.manager [instance: 604b621c-2103-4343-85f4-acaef2b0eb18]     preserve_ephemeral)
  2014-12-19 16:52:59.197 31679 TRACE nova.compute.manager [instance: 604b621c-2103-4343-85f4-acaef2b0eb18]   File "/opt/stack/new/nova/nova/virt/ironic/driver.py", line 297, in _add_driver_fields
  2014-12-19 16:52:59.197 31679 TRACE nova.compute.manager [instance: 604b621c-2103-4343-85f4-acaef2b0eb18]     ironicclient.call('node.update', node.uuid, patch)
  2014-12-19 16:52:59.197 31679 TRACE nova.compute.manager [instance: 604b621c-2103-4343-85f4-acaef2b0eb18]   File "/opt/stack/new/nova/nova/virt/ironic/client_wrapper.py", line 118, in call
  2014-12-19 16:52:59.197 31679 TRACE nova.compute.manager [instance: 604b621c-2103-4343-85f4-acaef2b0eb18]     raise exception.NovaException(msg)
  2014-12-19 16:52:59.197 31679 TRACE nova.compute.manager [instance: 604b621c-2103-4343-85f4-acaef2b0eb18] NovaException: Error contacting Ironic server for 'node.update'. Attempt 60 of 60
  2014-12-19 16:52:59.197 31679 TRACE nova.compute.manager [instance: 604b621c-2103-4343-85f4-acaef2b0eb18]

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1404331/+subscriptions