← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #01351

[Bug 1075320] Re: Instance deletion hangs after _do_terminate_instance

  Thread Previous • Date Previous • • Thread Next 
This bug lacks the necessary information to effectively reproduce and
fix it, therefore it has been closed. Feel free to reopen the bug by
providing the requested information and set the bug status back to
''New''.

** Changed in: nova
       Status: Incomplete => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1075320

Title:
  Instance deletion hangs after _do_terminate_instance

Status in Ceilometer:
  Confirmed
Status in OpenStack Compute (Nova):
  Invalid

Bug description:
  So when I have the below enabled and I do terminate in horizon on the
  given VM it fails.

  However if I restart nova-compute after commenting out the given
  options it works fine.

  Snippet from the Compute logs that the VM is running on.

  This happens when I have:
   71 instance_usage_audit = True
   72 instance_usage_audit_period = hour
   73 notification_driver=nova.openstack.common.notifier.rabbit_notifier
   74 notification_driver=ceilometer.compute.nova_notifier

  
  2012-11-05 22:02:04 DEBUG nova.openstack.common.rpc.amqp [-] received {u'_context_roles': [u'Member', u'admin'], u'_context_request_id': u'req-b71fb34b-6027-4725-88a0-9419d1e44d4e', u'_context_quota_class': None, u'_context_project_name': u'Smart', u'_context_service_cat
  alog': [{u'endpoints': [{u'adminURL': u'http://nova.os.lan:8774/v2/c97027dd880d4c129ae7a4ba7edade05', u'region': u'RegionOne', u'internalURL': u'http://nova.os.lan:8774/v2/c97027dd880d4c129ae7a4ba7edade05', u'id': u'7c348a9ac4744cedad5d13c6dd2617a5', u'publicURL': u'http
  ://nova.bouvetlab.local:8774/v2/c97027dd880d4c129ae7a4ba7edade05'}], u'endpoints_links': [], u'type': u'compute', u'name': u'nova'}, {u'endpoints': [{u'adminURL': u'http://quantum.os.lan:9696/', u'region': u'RegionOne', u'internalURL': u'http://quantum.os.lan:9696/', u'i
  d': u'02bbf0add27040978d591f53c3ffedf0', u'publicURL': u'http://quantum.bouvetlab.local:9696/'}], u'endpoints_links': [], u'type': u'network', u'name': u'quantum'}, {u'endpoints': [{u'adminURL': u'http://glance.os.lan:9292/v2', u'region': u'RegionOne', u'internalURL': u'
  http://glance.os.lan:9292/v2', u'id': u'c913e43a31744d499097234be00d5e6f', u'publicURL': u'http://glance.bouvetlab.local:9292/v2'}], u'endpoints_links': [], u'type': u'image', u'name': u'glance'}, {u'endpoints': [{u'adminURL': u'http://192.168.6.12:8777', u'region': u're
  gionOne', u'internalURL': u'http://192.168.6.12:8777', u'id': u'c72653c1f8044a15874eb27a5fae580e', u'publicURL': u'http://192.168.5.52:8777'}], u'endpoints_links': [], u'type': u'metering', u'name': u'ceilometer'}, {u'endpoints': [{u'adminURL': u'http://heat.os.lan:8000/
  v1', u'region': u'RegionOne', u'internalURL': u'http://heat.os.lan:8000/v1', u'id': u'bb5392c96c274710bdcd8646e2510c8b', u'publicURL': u'http://heat.os.lan:8000/v1'}], u'endpoints_links': [], u'type': u'cloudformation', u'name': u'heat'}, {u'endpoints': [{u'adminURL': u'
  http://cinder.os.lan:8776/v1/c97027dd880d4c129ae7a4ba7edade05', u'region': u'RegionOne', u'internalURL': u'http://cinder.os.lan:8776/v1/c97027dd880d4c129ae7a4ba7edade05', u'id': u'e83311fd966c440384580df55cc5b5c4', u'publicURL': u'http://cinder.bouvetlab.local:8776/v1/c9
  7027dd880d4c129ae7a4ba7edade05'}], u'endpoints_links': [], u'type': u'volume', u'name': u'cinder'}, {u'endpoints': [{u'adminURL': u'http://nova.os.lan:8773/services/Admin', u'region': u'RegionOne', u'internalURL': u'http://nova.os.lan:8773/services/Cloud', u'id': u'46d5d
  9e4ec9d41cb8a2ea8b439cdd2ee', u'publicURL': u'http://nova.bouvetlab.local:8773/services/Cloud'}], u'endpoints_links': [], u'type': u'ec2', u'name': u'ec2'}, {u'endpoints': [{u'adminURL': u'http://swift.os.lan:8080/v1', u'region': u'RegionOne', u'internalURL': u'http://sw
  ift.os.lan:8080/v1/AUTH_c97027dd880d4c129ae7a4ba7edade05', u'id': u'5f166002b1044286bc415b4fc89fdda8', u'publicURL': u'http://swift.bouvetlab.local:8080/v1/AUTH_c97027dd880d4c129ae7a4ba7edade05'}], u'endpoints_links': [], u'type': u'object-store', u'name': u'swift'}, {u'
  endpoints': [{u'adminURL': u'http://keystone.os.lan:35357/v2.0', u'region': u'RegionOne', u'internalURL': u'http://keystone.os.lan:5000/v2.0', u'id': u'788739bcbd06494785db65613ae2a289', u'publicURL': u'http://keystone.bouvetlab.local:5000/v2.0'}], u'endpoints_links': []
  , u'type': u'identity', u'name': u'keystone'}], u'_context_user_name': u'endre', u'_context_auth_token': '<SANITIZED>', u'args': {u'instance': {u'vm_state': u'active', u'availability_zone': None, u'terminated_at': None, u'ephemeral_gb': 20, u'instance_type_id': 5, u'user
  _data': None, u'vm_mode': None, u'deleted_at': None, u'reservation_id': u'r-pb0qj603', u'id': 335, u'security_groups': [{u'deleted_at': None, u'user_id': u'4ce5c085e09a478ea4edcd667a92df78', u'name': u'default', u'deleted': False, u'created_at': u'2012-10-11T10:00:59.000
  000', u'updated_at': None, u'rules': [{u'from_port': -1, u'protocol': u'icmp', u'group_id': None, u'deleted': False, u'created_at': u'2012-10-12T22:49:50.000000', u'updated_at': None, u'to_port': -1, u'parent_group_id': 53, u'cidr': u'0.0.0.0/0', u'deleted_at': None, u'i
  d': 15}, {u'from_port': 22, u'protocol': u'tcp', u'group_id': None, u'deleted': False, u'created_at': u'2012-10-15T10:32:51.000000', u'updated_at': None, u'to_port': 22, u'parent_group_id': 53, u'cidr': u'0.0.0.0/0', u'deleted_at': None, u'id': 16}, {u'from_port': 3389, u'protocol': u'tcp', u'group_id': None, u'deleted': False, u'created_at': u'2012-10-18T12:05:22.000000', u'updated_at': None, u'to_port': 3389, u'parent_group_id': 53, u'cidr': u'0.0.0.0/0', u'deleted_at': None, u'id': 17}, {u'from_port': 1099, u'protocol': u'tcp', u'group_id': None, u'deleted': False, u'created_at': u'2012-10-23T07:48:31.000000', u'updated_at': None, u'to_port': 1099, u'parent_group_id': 53, u'cidr': u'0.0.0.0/0', u'deleted_at': None, u'id': 18}, {u'from_port': 1, u'protocol': u'tcp', u'group_id': None, u'deleted': False, u'created_at': u'2012-10-23T10:59:25.000000', u'updated_at': None, u'to_port': 65535, u'parent_group_id': 53, u'cidr': u'0.0.0.0/0', u'deleted_at': None, u'id': 19}], u'project_id': u'c97027dd880d4c129ae7a4ba7edade05', u'id': 53, u'description': u'default'}], u'disable_terminate': False, u'user_id': u'4ce5c085e09a478ea4edcd667a92df78', u'uuid': u'8ea1adc7-b3af-4892-8fa8-3c8e76f5dd68', u'server_name': None, u'default_swap_device': None, u'info_cache': {u'instance_uuid': u'8ea1adc7-b3af-4892-8fa8-3c8e76f5dd68', u'deleted': True, u'created_at': u'2012-11-05T20:42:22.000000', u'updated_at': u'2012-11-05T20:57:55.000000', u'network_info': u'[{"network": {"bridge": "", "subnets": [{"ips": [{"meta": {}, "version": 4, "type": "fixed", "floating_ips": [], "address": "172.16.60.3"}], "version": 4, "meta": {"dhcp_server": "172.16.60.2"}, "dns": [], "routes": [], "cidr": "172.16.60.0/24", "gateway": {"meta": {}, "version": 4, "type": "gateway", "address": "172.16.60.1"}}], "meta": {"injected": false, "tenant_id": "c97027dd880d4c129ae7a4ba7edade05"}, "id": "8375aea7-bd6e-4a09-9e77-acb6494a1bbf", "label": "smart-net"}, "meta": {}, "id": "6888bc45-f8fb-461b-bb94-8e5633ab3239", "address": "fa:16:3e:1f:a0:63"}]', u'deleted_at': u'2012-11-05T20:57:55.000000', u'id': 335}, u'hostname': u'test', u'launched_on': u'cmp01', u'display_description': u'test', u'key_data': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC3c+DgMce8RjzD+2DtC2L8h/4VpmGKURKl6UlVmjiMGpZ/y7/hdqfu165xn5hxcQYmpQqnYUeUkuETTWViVJtZGHmkroM7pjVY9Uaaztndk1YsNduNs4OmIV5wGgjTQKMQNqOU21bygzcxwFNOzM0tH1LoDhco+daSbGFdhtzjENvPKBF3234sobJFPfuuDgJ1GSY16+UrYYwVmvRaK1SQUffFQT6jGTnb7Az7n/6siugy18nULKI5dNB7XGDSnKSWT8WftnzuuQShukaUzZM9tH3BLU8aqc+3112jWP6XctDKOGed2QpemifcQFchzVj9srOUqsq4fVugrAWVqSU/ client@admin', u'kernel_id': u'', u'power_state': 1, u'default_ephemeral_device': u'/dev/vdb', u'progress': 0, u'project_id': u'c97027dd880d4c129ae7a4ba7edade05', u'launched_at': u'2012-11-05T20:43:22.000000', u'scheduled_at': u'2012-11-05T20:42:22.000000', u'ramdisk_id': u'', u'access_ip_v6': None, u'access_ip_v4': None, u'deleted': False, u'key_name': u'default', u'updated_at': u'2012-11-05T20:57:55.000000', u'host': u'cmp01', u'display_name': u'test', u'task_state': u'deleting', u'shutdown_terminate': False, u'architecture': None, u'root_gb': 10, u'locked': False, u'name': u'vm-0000014f', u'created_at': u'2012-11-05T20:42:22.000000', u'launch_index': 0, u'metadata': [], u'memory_mb': 2048, u'instance_type': {u'disabled': False, u'root_gb': 10, u'deleted_at': None, u'name': u'm1.small', u'deleted': False, u'created_at': None, u'ephemeral_gb': 20, u'updated_at': None, u'memory_mb': 2048, u'vcpus': 1, u'swap': 0, u'rxtx_factor': 1.0, u'is_public': True, u'flavorid': u'2', u'vcpu_weight': None, u'id': 5}, u'vcpus': 1, u'image_ref': u'0906f07d-f419-4a9d-8404-047db4717497', u'root_device_name': u'/dev/vda', u'auto_disk_config': None, u'os_type': None, u'config_drive': u''}}, u'_context_instance_lock_checked': False, u'_context_is_admin': False, u'version': u'2.0', u'_context_project_id': u'c97027dd880d4c129ae7a4ba7edade05', u'_context_timestamp': u'2012-11-05T21:01:48.753541', u'_context_read_deleted': u'no', u'_context_user_id': u'4ce5c085e09a478ea4edcd667a92df78', u'method': u'terminate_instance', u'_context_remote_address': u'192.168.5.51'} from (pid=33107) _safe_log /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/common.py:195
  2012-11-05 22:02:04 DEBUG nova.openstack.common.rpc.amqp [-] unpacked context: {'project_name': u'Smart', 'user_id': u'4ce5c085e09a478ea4edcd667a92df78', 'roles': [u'Member', u'admin'], 'timestamp': u'2012-11-05T21:01:48.753541', 'auth_token': '<SANITIZED>', 'remote_address': u'192.168.5.51', 'quota_class': None, 'is_admin': False, 'service_catalog': [{u'endpoints_links': [], u'endpoints': [{u'adminURL': u'http://nova.os.lan:8774/v2/c97027dd880d4c129ae7a4ba7edade05', u'region': u'RegionOne', u'publicURL': u'http://nova.bouvetlab.local:8774/v2/c97027dd880d4c129ae7a4ba7edade05', u'internalURL': u'http://nova.os.lan:8774/v2/c97027dd880d4c129ae7a4ba7edade05', u'id': u'7c348a9ac4744cedad5d13c6dd2617a5'}], u'type': u'compute', u'name': u'nova'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'http://quantum.os.lan:9696/', u'region': u'RegionOne', u'publicURL': u'http://quantum.bouvetlab.local:9696/', u'internalURL': u'http://quantum.os.lan:9696/', u'id': u'02bbf0add27040978d591f53c3ffedf0'}], u'type': u'network', u'name': u'quantum'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'http://glance.os.lan:9292/v2', u'region': u'RegionOne', u'publicURL': u'http://glance.bouvetlab.local:9292/v2', u'internalURL': u'http://glance.os.lan:9292/v2', u'id': u'c913e43a31744d499097234be00d5e6f'}], u'type': u'image', u'name': u'glance'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'http://192.168.6.12:8777', u'region': u'regionOne', u'publicURL': u'http://192.168.5.52:8777', u'internalURL': u'http://192.168.6.12:8777', u'id': u'c72653c1f8044a15874eb27a5fae580e'}], u'type': u'metering', u'name': u'ceilometer'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'http://heat.os.lan:8000/v1', u'region': u'RegionOne', u'publicURL': u'http://heat.os.lan:8000/v1', u'internalURL': u'http://heat.os.lan:8000/v1', u'id': u'bb5392c96c274710bdcd8646e2510c8b'}], u'type': u'cloudformation', u'name': u'heat'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'http://cinder.os.lan:8776/v1/c97027dd880d4c129ae7a4ba7edade05', u'region': u'RegionOne', u'publicURL': u'http://cinder.bouvetlab.local:8776/v1/c97027dd880d4c129ae7a4ba7edade05', u'internalURL': u'http://cinder.os.lan:8776/v1/c97027dd880d4c129ae7a4ba7edade05', u'id': u'e83311fd966c440384580df55cc5b5c4'}], u'type': u'volume', u'name': u'cinder'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'http://nova.os.lan:8773/services/Admin', u'region': u'RegionOne', u'publicURL': u'http://nova.bouvetlab.local:8773/services/Cloud', u'internalURL': u'http://nova.os.lan:8773/services/Cloud', u'id': u'46d5d9e4ec9d41cb8a2ea8b439cdd2ee'}], u'type': u'ec2', u'name': u'ec2'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'http://swift.os.lan:8080/v1', u'region': u'RegionOne', u'publicURL': u'http://swift.bouvetlab.local:8080/v1/AUTH_c97027dd880d4c129ae7a4ba7edade05', u'internalURL': u'http://swift.os.lan:8080/v1/AUTH_c97027dd880d4c129ae7a4ba7edade05', u'id': u'5f166002b1044286bc415b4fc89fdda8'}], u'type': u'object-store', u'name': u'swift'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'http://keystone.os.lan:35357/v2.0', u'region': u'RegionOne', u'publicURL': u'http://keystone.bouvetlab.local:5000/v2.0', u'internalURL': u'http://keystone.os.lan:5000/v2.0', u'id': u'788739bcbd06494785db65613ae2a289'}], u'type': u'identity', u'name': u'keystone'}], 'request_id': u'req-b71fb34b-6027-4725-88a0-9419d1e44d4e', 'instance_lock_checked': False, 'project_id': u'c97027dd880d4c129ae7a4ba7edade05', 'user_name': u'endre', 'read_deleted': u'no'} from (pid=33107) _safe_log /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/common.py:195
  2012-11-05 22:02:04 DEBUG nova.utils [req-b71fb34b-6027-4725-88a0-9419d1e44d4e 4ce5c085e09a478ea4edcd667a92df78 c97027dd880d4c129ae7a4ba7edade05] Got semaphore "8ea1adc7-b3af-4892-8fa8-3c8e76f5dd68" for method "do_terminate_instance"... from (pid=33107) inner /usr/lib/python2.7/dist-packages/nova/utils.py:713
  2012-11-05 22:02:04 DEBUG nova.openstack.common.rpc.amqp [-] Sending compute.instance.delete.start on notifications.info from (pid=33107) notify /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py:410
  2012-11-05 22:02:04 DEBUG nova.openstack.common.rpc.amqp [-] Pool creating new connection from (pid=33107) create /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py:57
  2012-11-05 22:02:04 INFO nova.openstack.common.rpc.common [-] Connected to AMQP server on nova.os.lan:5672
  2012-11-05 22:02:04 DEBUG nova.ceilometer.notifier [-] polling final stats for u'8ea1adc7-b3af-4892-8fa8-3c8e76f5dd68' from (pid=33107) notify /opt/ceilometer/ceilometer/compute/nova_notifier.py:62
  2012-11-05 22:02:04 DEBUG nova.virt.libvirt.driver [req-b71fb34b-6027-4725-88a0-9419d1e44d4e 4ce5c085e09a478ea4edcd667a92df78 c97027dd880d4c129ae7a4ba7edade05] Connecting to libvirt: qemu:///system from (pid=33107) _get_connection /usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py:340
    2012-11-05 22:02:48 DEBUG nova.manager [req-9f2ef547-c3a5-4be2-b4df-de9f402c7232 None None] Running periodic task ComputeManager._publish_service_capabilities from (pid=33107) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:172
  2012-11-05 22:02:48 DEBUG nova.manager [req-9f2ef547-c3a5-4be2-b4df-de9f402c7232 None None] Running periodic task ComputeManager._poll_rescued_instances from (pid=33107) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:172
  2012-11-05 22:02:48 DEBUG nova.manager [req-9f2ef547-c3a5-4be2-b4df-de9f402c7232 None None] Skipping ComputeManager._sync_power_states, 10 ticks left until next run from (pid=33107) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:167
  2012-11-05 22:02:48 DEBUG nova.manager [req-9f2ef547-c3a5-4be2-b4df-de9f402c7232 None None] Running periodic task ComputeManager._poll_bandwidth_usage from (pid=33107) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:172

To manage notifications about this bug go to:
https://bugs.launchpad.net/ceilometer/+bug/1075320/+subscriptions
  Thread Previous • Date Previous • • Thread Next