← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 1477490] Re: Ironic: Deleting while spawning can leave orphan ACTIVE nodes in Ironic

 

Hi all!

It seems that the Nova fix has landed, and now we at least make sure
that the deletion does not succeed for DEPLOYING nodes. I think it's
enough to mark this bug as fixed. I suggest filing a new RFE for the
ability to delete nodes in any state, as the previously filed spec is
not abandoned. Thanks!

** Changed in: nova
       Status: Confirmed => Fix Released

** Changed in: ironic
       Status: Confirmed => Won't Fix

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1477490

Title:
  Ironic: Deleting while spawning can leave orphan ACTIVE nodes in
  Ironic

Status in Ironic:
  Won't Fix
Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  The Ironic nova driver won't try to delete the instance in Ironic if
  the node's provision state is DEPLOYING [1] , this is known to fail
  with the current Ironic code because we just can't abort the
  installation at the DEPLOYING stage.

  But the Ironic nova driver just keep going and tries to clean up the
  deployment environment (without telling Ironic to unprovision the
  instance) and it will fail as well. But the the code that cleans up
  the instance will keep retrying [3] because there's a transition in
  progress and it can't update the node. But when the node finishes the
  deployment, if the retrying didn't timed out, the destroy() method
  from the Nova driver  will succeed cleaning deployment environment and
  the Nova instance will be deleted but the Ironic node will continue to
  marked as ACTIVE in Ironic and now orphan because there's no instance
  in Nova associated with it [4]

  The good news is that since nova clean up the network stuff the
  instance won't be accessible.

  WORKAROUND:

  Unprovision the node using the Ironic API directly

  $ ironic node-set-provision-state <node uuid> deleted

  PROPOSED FIX:

  IMO the ironic nova driver should try to tell Ironic to delete the
  instance even when the provision state of the node is DEPLOYING. If it
  fails the nova delete command will fail saying it can not delete the
  instance, which is fine until this gets resolved in Ironic (there's
  work going on to be able to abort a deployment at any stage)

  
  [1] https://github.com/openstack/nova/blob/6a24bbeecd8a6d6d3135a10f4917b071896d14ee/nova/virt/ironic/driver.py#L865-L868

  [2]
  https://github.com/openstack/nova/blob/6a24bbeecd8a6d6d3135a10f4917b071896d14ee/nova/virt/ironic/driver.py#L871

  [3] From the nova-compute logs

  {"error_message": "{\"debuginfo\": null, \"faultcode\": \"Client\", \"faultstring\": \"Node d240ae0d-1844-48f0-adcf-b70680a1b6ce can not be updated while a state transition is in progress.\"}"}
   from (pid=6672) log_http_response /usr/local/lib/python2.7/dist-packages/ironicclient/common/http.py:260
  2015-07-23 11:07:40.358 WARNING ironicclient.common.http [req-24b39fe8-435d-4869-970f-53f64b3512a8 demo demo] Request returned failure status.
  2015-07-23 11:07:40.358 WARNING ironicclient.common.http [req-24b39fe8-435d-4869-970f-53f64b3512a8 demo demo] Error contacting Ironic server: Node d240ae0d-1844-48f0-adcf-b70680a1b6ce can not be updated while a state transition is in progress. (HTTP 409). Attempt 3 of 6

  [4] http://paste.openstack.org/show/403569/

To manage notifications about this bug go to:
https://bugs.launchpad.net/ironic/+bug/1477490/+subscriptions


References