← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #26442

[Bug 1405107] [NEW] The instance's task_state is "deleting" all the time, because the error of "Instance disappeared during terminate".

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

A instance's vm_state is error.  I did delete it, but the task_state is
"deleting" all the time. The log show that the error is "Instance
disappeared during terminate" because of InstanceNotFound exception.

1. Relative code is as follows:
@utils.synchronized(instance['uuid'])
        def do_terminate_instance(instance, bdms):
            try:
                self._delete_instance(context, instance, bdms, quotas)
            except exception.InstanceNotFound:
                LOG.info(_("Instance disappeared during terminate"),
                         instance=instance)
            except Exception as error:
                # As we're trying to delete always go to Error if something
                # goes wrong that _delete_instance can't handle.
                with excutils.save_and_reraise_exception():
                    LOG.exception(_('Setting instance vm_state to ERROR'),
                                  instance=instance)
                    self._set_instance_error_state(context, instance['uuid'])


2. [root@lxlconductor1 instances(keystone_admin)]# nova list
+--------------------------------------+----------+--------+------------+-------------+-------------------+
| ID                                   | Name     | Status | Task State | Power State | Networks          |
+--------------------------------------+----------+--------+------------+-------------+-------------------+
| e72d62c9-5d54-4bd0-8afc-d45116467ba5 | hanrong2 | ACTIVE | deleting   | NOSTATE     |                   |
+--------------------------------------+----------+--------+------------+-------------+-------------------+

3. The error log is as follows:
2015-02-21 17:20:41.049 429 AUDIT nova.compute.manager [req-621830f5-8d73-4897-90f9-505e4bf9a6e4 7aba40236a4c4494aba4eb0b9365ffee be568b8239d147e58f1ef16b6011c93d] [instance: e72d62c9-5d54-4bd0-8afc-d45116467ba5] Terminating instance
2015-02-21 17:21:06.031 429 ERROR nova.virt.libvirt.driver [-] [instance: e72d62c9-5d54-4bd0-8afc-d45116467ba5] During wait destroy, instance disappeared.
2015-02-21 17:21:06.066 429 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 10.43.114.108
2015-02-21 17:21:06.550 429 INFO nova.virt.libvirt.driver [req-621830f5-8d73-4897-90f9-505e4bf9a6e4 7aba40236a4c4494aba4eb0b9365ffee be568b8239d147e58f1ef16b6011c93d] [instance: e72d62c9-5d54-4bd0-8afc-d45116467ba5] Deletion of /var/lib/nova/instances/e72d62c9-5d54-4bd0-8afc-d45116467ba5 complete
2015-02-21 17:21:38.469 429 AUDIT nova.compute.resource_tracker [req-5aae7032-2766-4adf-be17-ab059b4d5cae None None] Auditing locally available compute resources
2015-02-21 17:21:38.612 429 INFO oslo.messaging._drivers.impl_qpid [-] Connected to AMQP server on 10.43.114.108:5671
2015-02-21 17:22:44.876 429 INFO oslo.messaging._drivers.impl_qpid [-] Connected to AMQP server on 10.43.114.108:5671
2015-02-21 17:23:05.684 429 INFO nova.compute.manager [req-621830f5-8d73-4897-90f9-505e4bf9a6e4 7aba40236a4c4494aba4eb0b9365ffee be568b8239d147e58f1ef16b6011c93d] [instance: e72d62c9-5d54-4bd0-8afc-d45116467ba5] Instance disappeared during terminate

4. I think if instance is disappeared, delete would be successful.

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1405107

Title:
  The instance's task_state is "deleting" all the time, because the
  error of "Instance disappeared during terminate".

Status in OpenStack Compute (Nova):
  New

Bug description:
  A instance's vm_state is error.  I did delete it, but the task_state
  is "deleting" all the time. The log show that the error is "Instance
  disappeared during terminate" because of InstanceNotFound exception.

  1. Relative code is as follows:
  @utils.synchronized(instance['uuid'])
          def do_terminate_instance(instance, bdms):
              try:
                  self._delete_instance(context, instance, bdms, quotas)
              except exception.InstanceNotFound:
                  LOG.info(_("Instance disappeared during terminate"),
                           instance=instance)
              except Exception as error:
                  # As we're trying to delete always go to Error if something
                  # goes wrong that _delete_instance can't handle.
                  with excutils.save_and_reraise_exception():
                      LOG.exception(_('Setting instance vm_state to ERROR'),
                                    instance=instance)
                      self._set_instance_error_state(context, instance['uuid'])

  
  2. [root@lxlconductor1 instances(keystone_admin)]# nova list
  +--------------------------------------+----------+--------+------------+-------------+-------------------+
  | ID                                   | Name     | Status | Task State | Power State | Networks          |
  +--------------------------------------+----------+--------+------------+-------------+-------------------+
  | e72d62c9-5d54-4bd0-8afc-d45116467ba5 | hanrong2 | ACTIVE | deleting   | NOSTATE     |                   |
  +--------------------------------------+----------+--------+------------+-------------+-------------------+

  3. The error log is as follows:
  2015-02-21 17:20:41.049 429 AUDIT nova.compute.manager [req-621830f5-8d73-4897-90f9-505e4bf9a6e4 7aba40236a4c4494aba4eb0b9365ffee be568b8239d147e58f1ef16b6011c93d] [instance: e72d62c9-5d54-4bd0-8afc-d45116467ba5] Terminating instance
  2015-02-21 17:21:06.031 429 ERROR nova.virt.libvirt.driver [-] [instance: e72d62c9-5d54-4bd0-8afc-d45116467ba5] During wait destroy, instance disappeared.
  2015-02-21 17:21:06.066 429 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 10.43.114.108
  2015-02-21 17:21:06.550 429 INFO nova.virt.libvirt.driver [req-621830f5-8d73-4897-90f9-505e4bf9a6e4 7aba40236a4c4494aba4eb0b9365ffee be568b8239d147e58f1ef16b6011c93d] [instance: e72d62c9-5d54-4bd0-8afc-d45116467ba5] Deletion of /var/lib/nova/instances/e72d62c9-5d54-4bd0-8afc-d45116467ba5 complete
  2015-02-21 17:21:38.469 429 AUDIT nova.compute.resource_tracker [req-5aae7032-2766-4adf-be17-ab059b4d5cae None None] Auditing locally available compute resources
  2015-02-21 17:21:38.612 429 INFO oslo.messaging._drivers.impl_qpid [-] Connected to AMQP server on 10.43.114.108:5671
  2015-02-21 17:22:44.876 429 INFO oslo.messaging._drivers.impl_qpid [-] Connected to AMQP server on 10.43.114.108:5671
  2015-02-21 17:23:05.684 429 INFO nova.compute.manager [req-621830f5-8d73-4897-90f9-505e4bf9a6e4 7aba40236a4c4494aba4eb0b9365ffee be568b8239d147e58f1ef16b6011c93d] [instance: e72d62c9-5d54-4bd0-8afc-d45116467ba5] Instance disappeared during terminate

  4. I think if instance is disappeared, delete would be successful.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1405107/+subscriptions

Follow ups

References

  Thread Previous • Date Previous • • Thread Next