yahoo-eng-team team mailing list archive

[Sean Dague <sean@xxxxxxxxx>]

** No longer affects: nova/folsom

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1110221

Title:
  Instance stuck in ACTIVE/deleting when network deallocation fails

Status in OpenStack Compute (Nova):
  Fix Released

Bug description:
  I set a simple bash script that boots a VM, waits a couple of minutes
  (doing nova list) then deletes the VM, waits a bit and continues.

  I ran this last night, and one of the VMs has not been deleted. Log is
  showing that the delete was seen and processed without error by the
  API node but in compute log there is timeout while deallocating
  network for an instance.

  ERROR nova.openstack.common.rpc.common [-] Timed out waiting for RPC response: timed out
  TRACE nova.openstack.common.rpc.common Traceback (most recent call last):
  TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 513, in ensure
  TRACE nova.openstack.common.rpc.common return method(*args, **kwargs)
  TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 590, in _consume
  TRACE nova.openstack.common.rpc.common return self.connection.drain_events(timeout=timeout)
  TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 175, in drain_events
  TRACE nova.openstack.common.rpc.common return self.transport.drain_events(self.connection, **kwargs)
  TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 238, in drain_events
  TRACE nova.openstack.common.rpc.common return connection.drain_events(**kwargs)
  TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 57, in drain_events
  TRACE nova.openstack.common.rpc.common return self.wait_multi(self.channels.values(), timeout=timeout)
  TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 63, in wait_multi
  TRACE nova.openstack.common.rpc.common chanmap.keys(), allowed_methods, timeout=timeout)
  TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 120, in _wait_multiple
  TRACE nova.openstack.common.rpc.common channel, method_sig, args, content = read_timeout(timeout)
  TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 94, in read_timeout
  TRACE nova.openstack.common.rpc.common return self.method_reader.read_method()
  TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/method_framing.py", line 221, in read_method
  TRACE nova.openstack.common.rpc.common raise m
  TRACE nova.openstack.common.rpc.common timeout: timed out
  TRACE nova.openstack.common.rpc.common

  When timeout exception is raised during network deallocation then
  instances stuck in Active(deleting)  but as nova list displays only
  vm_state, VM will get displayed as ACTIVE and there is no periodic
  task which cleans-up instances stuck in Active(deleting) on service
  restart..

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1110221/+subscriptions