yahoo-eng-team team mailing list archive

[Evgeniy Afonichev <evgeniy@xxxxxxxxxxxxx>]

couldn't reproduce. Also I suspect that volume attach/detach led to db
inconsistencies

** Changed in: nova
       Status: Incomplete => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1426524

Title:
  race condition prevents intance deletion

Status in OpenStack Compute (Nova):
  Invalid

Bug description:
  Version: icehouse. Though looking in to the code in the master I believe bug is still there
  Hypervisor: libvirt
  Frequency: very rare, under heavy load (stress tests)
  Steps to reproduce: as an operator I issue "nova delete" command. Instead of being deleted that vm gets into ERROR state.

  relevant nova-compute.log: http://paste.openstack.org/show/183111/

  Here's probably why it happens:
  It's a race condition. There are two threads (coroutines if eventlet patched) - thread-1 which handles termination request (nova.compute.manager.ComputeManager.terminate_instance) and thread-2 which dispatches events from hypervisor.
  1) thread-1: manager clears (deletes) all queued events for that vm and switches to thread-2 https://github.com/openstack/nova/blob/983f755562cb87a0b498af5d62be9bd2010bc999/nova/compute/manager.py#L2526
  2) thread-2: hypervisor emits one more event and switches to thread-1 without dispatching event
  3) thread-1: manager deletes image files, marks instance as deleted in the db. Thread finishes and exits normally
  4) thread-2: manager tries to dispatch one more event. But fails as there is no such instance anymore. To be more precise - there is no InstanceInfoCache for that vm.

  
  UPD: more logs: https://www.dropbox.com/sh/r0ek3w7g95qoetw/AADTfgN9tD2Mt_fOXjB9OCzva?dl=0
  - Cluster is in HA mode (three nova-api files)
  - debug=False
  - conductor-logs contain only "connecting, reconnecting to rabbitmq". 
  - cat ... | grep -A 15 -B 15 -r 09ac0ed2-07cb-4394-b0c8-aff3ab74dcdb

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1426524/+subscriptions