yahoo-eng-team team mailing list archive

[Joe Gordon <1405107@xxxxxxxxxxxxxxxxxx>]

Having an instance disappear isn't enough to know the things were
cleaned up successfully. Instead we should sort out why the instance
disappeared, that shouldn't happen.

** Changed in: nova
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1405107

Title:
  The instance's task_state is "deleting" all the time, because the
  error of "Instance disappeared during terminate".

Status in OpenStack Compute (Nova):
  Invalid

Bug description:
  A instance's vm_state is error.  I did delete it, but the task_state
  is "deleting" all the time. The log show that the error is "Instance
  disappeared during terminate" because of InstanceNotFound exception.

  1. Relative code is as follows:
  @utils.synchronized(instance['uuid'])
          def do_terminate_instance(instance, bdms):
              try:
                  self._delete_instance(context, instance, bdms, quotas)
              except exception.InstanceNotFound:
                  LOG.info(_("Instance disappeared during terminate"),
                           instance=instance)
              except Exception as error:
                  # As we're trying to delete always go to Error if something
                  # goes wrong that _delete_instance can't handle.
                  with excutils.save_and_reraise_exception():
                      LOG.exception(_('Setting instance vm_state to ERROR'),
                                    instance=instance)
                      self._set_instance_error_state(context, instance['uuid'])

  
  2. [root@lxlconductor1 instances(keystone_admin)]# nova list
  +--------------------------------------+----------+--------+------------+-------------+-------------------+
  | ID                                   | Name     | Status | Task State | Power State | Networks          |
  +--------------------------------------+----------+--------+------------+-------------+-------------------+
  | e72d62c9-5d54-4bd0-8afc-d45116467ba5 | hanrong2 | ACTIVE | deleting   | NOSTATE     |                   |
  +--------------------------------------+----------+--------+------------+-------------+-------------------+

  3. The error log is as follows:
  2015-02-21 17:20:41.049 429 AUDIT nova.compute.manager [req-621830f5-8d73-4897-90f9-505e4bf9a6e4 7aba40236a4c4494aba4eb0b9365ffee be568b8239d147e58f1ef16b6011c93d] [instance: e72d62c9-5d54-4bd0-8afc-d45116467ba5] Terminating instance
  2015-02-21 17:21:06.031 429 ERROR nova.virt.libvirt.driver [-] [instance: e72d62c9-5d54-4bd0-8afc-d45116467ba5] During wait destroy, instance disappeared.
  2015-02-21 17:21:06.066 429 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 10.43.114.108
  2015-02-21 17:21:06.550 429 INFO nova.virt.libvirt.driver [req-621830f5-8d73-4897-90f9-505e4bf9a6e4 7aba40236a4c4494aba4eb0b9365ffee be568b8239d147e58f1ef16b6011c93d] [instance: e72d62c9-5d54-4bd0-8afc-d45116467ba5] Deletion of /var/lib/nova/instances/e72d62c9-5d54-4bd0-8afc-d45116467ba5 complete
  2015-02-21 17:21:38.469 429 AUDIT nova.compute.resource_tracker [req-5aae7032-2766-4adf-be17-ab059b4d5cae None None] Auditing locally available compute resources
  2015-02-21 17:21:38.612 429 INFO oslo.messaging._drivers.impl_qpid [-] Connected to AMQP server on 10.43.114.108:5671
  2015-02-21 17:22:44.876 429 INFO oslo.messaging._drivers.impl_qpid [-] Connected to AMQP server on 10.43.114.108:5671
  2015-02-21 17:23:05.684 429 INFO nova.compute.manager [req-621830f5-8d73-4897-90f9-505e4bf9a6e4 7aba40236a4c4494aba4eb0b9365ffee be568b8239d147e58f1ef16b6011c93d] [instance: e72d62c9-5d54-4bd0-8afc-d45116467ba5] Instance disappeared during terminate

  4. I think if instance is disappeared, delete would be successful.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1405107/+subscriptions