← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #02200

[Bug 1075320] Re: Instance deletion hangs after _do_terminate_instance

  Thread Previous • Date Previous • • Thread Next 
** Changed in: ceilometer
       Status: Confirmed => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1075320

Title:
  Instance deletion hangs after _do_terminate_instance

Status in Ceilometer:
  Invalid
Status in OpenStack Compute (Nova):
  Invalid

Bug description:
  So when I have the below enabled and I do terminate in horizon on the
  given VM it fails.

  However if I restart nova-compute after commenting out the given
  options it works fine.

  Snippet from the Compute logs that the VM is running on.

  This happens when I have:
   71 instance_usage_audit = True
   72 instance_usage_audit_period = hour
   73 notification_driver=nova.openstack.common.notifier.rabbit_notifier
   74 notification_driver=ceilometer.compute.nova_notifier

  
  2012-11-05 22:02:04 DEBUG nova.openstack.common.rpc.amqp [-] received {u'_context_roles': [u'Member', u'admin'], u'_context_request_id': u'req-b71fb34b-6027-4725-88a0-9419d1e44d4e', u'_context_quota_class': None, u'_context_project_name': u'Smart', u'_context_service_cat
  alog': [{u'endpoints': [{u'adminURL': u'http://nova.os.lan:8774/v2/c97027dd880d4c129ae7a4ba7edade05', u'region': u'RegionOne', u'internalURL': u'http://nova.os.lan:8774/v2/c97027dd880d4c129ae7a4ba7edade05', u'id': u'7c348a9ac4744cedad5d13c6dd2617a5', u'publicURL': u'http
  ://nova.bouvetlab.local:8774/v2/c97027dd880d4c129ae7a4ba7edade05'}], u'endpoints_links': [], u'type': u'compute', u'name': u'nova'}, {u'endpoints': [{u'adminURL': u'http://quantum.os.lan:9696/', u'region': u'RegionOne', u'internalURL': u'http://quantum.os.lan:9696/', u'i
  d': u'02bbf0add27040978d591f53c3ffedf0', u'publicURL': u'http://quantum.bouvetlab.local:9696/'}], u'endpoints_links': [], u'type': u'network', u'name': u'quantum'}, {u'endpoints': [{u'adminURL': u'http://glance.os.lan:9292/v2', u'region': u'RegionOne', u'internalURL': u'
  http://glance.os.lan:9292/v2', u'id': u'c913e43a31744d499097234be00d5e6f', u'publicURL': u'http://glance.bouvetlab.local:9292/v2'}], u'endpoints_links': [], u'type': u'image', u'name': u'glance'}, {u'endpoints': [{u'adminURL': u'http://192.168.6.12:8777', u'region': u're
  gionOne', u'internalURL': u'http://192.168.6.12:8777', u'id': u'c72653c1f8044a15874eb27a5fae580e', u'publicURL': u'http://192.168.5.52:8777'}], u'endpoints_links': [], u'type': u'metering', u'name': u'ceilometer'}, {u'endpoints': [{u'adminURL': u'http://heat.os.lan:8000/
  v1', u'region': u'RegionOne', u'internalURL': u'http://heat.os.lan:8000/v1', u'id': u'bb5392c96c274710bdcd8646e2510c8b', u'publicURL': u'http://heat.os.lan:8000/v1'}], u'endpoints_links': [], u'type': u'cloudformation', u'name': u'heat'}, {u'endpoints': [{u'adminURL': u'
  http://cinder.os.lan:8776/v1/c97027dd880d4c129ae7a4ba7edade05', u'region': u'RegionOne', u'internalURL': u'http://cinder.os.lan:8776/v1/c97027dd880d4c129ae7a4ba7edade05', u'id': u'e83311fd966c440384580df55cc5b5c4', u'publicURL': u'http://cinder.bouvetlab.local:8776/v1/c9
  7027dd880d4c129ae7a4ba7edade05'}], u'endpoints_links': [], u'type': u'volume', u'name': u'cinder'}, {u'endpoints': [{u'adminURL': u'http://nova.os.lan:8773/services/Admin', u'region': u'RegionOne', u'internalURL': u'http://nova.os.lan:8773/services/Cloud', u'id': u'46d5d
  9e4ec9d41cb8a2ea8b439cdd2ee', u'publicURL': u'http://nova.bouvetlab.local:8773/services/Cloud'}], u'endpoints_links': [], u'type': u'ec2', u'name': u'ec2'}, {u'endpoints': [{u'adminURL': u'http://swift.os.lan:8080/v1', u'region': u'RegionOne', u'internalURL': u'http://sw
  ift.os.lan:8080/v1/AUTH_c97027dd880d4c129ae7a4ba7edade05', u'id': u'5f166002b1044286bc415b4fc89fdda8', u'publicURL': u'http://swift.bouvetlab.local:8080/v1/AUTH_c97027dd880d4c129ae7a4ba7edade05'}], u'endpoints_links': [], u'type': u'object-store', u'name': u'swift'}, {u'
  endpoints': [{u'adminURL': u'http://keystone.os.lan:35357/v2.0', u'region': u'RegionOne', u'internalURL': u'http://keystone.os.lan:5000/v2.0', u'id': u'788739bcbd06494785db65613ae2a289', u'publicURL': u'http://keystone.bouvetlab.local:5000/v2.0'}], u'endpoints_links': []
  , u'type': u'identity', u'name': u'keystone'}], u'_context_user_name': u'endre', u'_context_auth_token': '<SANITIZED>', u'args': {u'instance': {u'vm_state': u'active', u'availability_zone': None, u'terminated_at': None, u'ephemeral_gb': 20, u'instance_type_id': 5, u'user
  _data': None, u'vm_mode': None, u'deleted_at': None, u'reservation_id': u'r-pb0qj603', u'id': 335, u'security_groups': [{u'deleted_at': None, u'user_id': u'4ce5c085e09a478ea4edcd667a92df78', u'name': u'default', u'deleted': False, u'created_at': u'2012-10-11T10:00:59.000
  000', u'updated_at': None, u'rules': [{u'from_port': -1, u'protocol': u'icmp', u'group_id': None, u'deleted': False, u'created_at': u'2012-10-12T22:49:50.000000', u'updated_at': None, u'to_port': -1, u'parent_group_id': 53, u'cidr': u'0.0.0.0/0', u'deleted_at': None, u'i
  d': 15}, {u'from_port': 22, u'protocol': u'tcp', u'group_id': None, u'deleted': False, u'created_at': u'2012-10-15T10:32:51.000000', u'updated_at': None, u'to_port': 22, u'parent_group_id': 53, u'cidr': u'0.0.0.0/0', u'deleted_at': None, u'id': 16}, {u'from_port': 3389, u'protocol': u'tcp', u'group_id': None, u'deleted': False, u'created_at': u'2012-10-18T12:05:22.000000', u'updated_at': None, u'to_port': 3389, u'parent_group_id': 53, u'cidr': u'0.0.0.0/0', u'deleted_at': None, u'id': 17}, {u'from_port': 1099, u'protocol': u'tcp', u'group_id': None, u'deleted': False, u'created_at': u'2012-10-23T07:48:31.000000', u'updated_at': None, u'to_port': 1099, u'parent_group_id': 53, u'cidr': u'0.0.0.0/0', u'deleted_at': None, u'id': 18}, {u'from_port': 1, u'protocol': u'tcp', u'group_id': None, u'deleted': False, u'created_at': u'2012-10-23T10:59:25.000000', u'updated_at': None, u'to_port': 65535, u'parent_group_id': 53, u'cidr': u'0.0.0.0/0', u'deleted_at': None, u'id': 19}], u'project_id': u'c97027dd880d4c129ae7a4ba7edade05', u'id': 53, u'description': u'default'}], u'disable_terminate': False, u'user_id': u'4ce5c085e09a478ea4edcd667a92df78', u'uuid': u'8ea1adc7-b3af-4892-8fa8-3c8e76f5dd68', u'server_name': None, u'default_swap_device': None, u'info_cache': {u'instance_uuid': u'8ea1adc7-b3af-4892-8fa8-3c8e76f5dd68', u'deleted': True, u'created_at': u'2012-11-05T20:42:22.000000', u'updated_at': u'2012-11-05T20:57:55.000000', u'network_info': u'[{"network": {"bridge": "", "subnets": [{"ips": [{"meta": {}, "version": 4, "type": "fixed", "floating_ips": [], "address": "172.16.60.3"}], "version": 4, "meta": {"dhcp_server": "172.16.60.2"}, "dns": [], "routes": [], "cidr": "172.16.60.0/24", "gateway": {"meta": {}, "version": 4, "type": "gateway", "address": "172.16.60.1"}}], "meta": {"injected": false, "tenant_id": "c97027dd880d4c129ae7a4ba7edade05"}, "id": "8375aea7-bd6e-4a09-9e77-acb6494a1bbf", "label": "smart-net"}, "meta": {}, "id": "6888bc45-f8fb-461b-bb94-8e5633ab3239", "address": "fa:16:3e:1f:a0:63"}]', u'deleted_at': u'2012-11-05T20:57:55.000000', u'id': 335}, u'hostname': u'test', u'launched_on': u'cmp01', u'display_description': u'test', u'key_data': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC3c+DgMce8RjzD+2DtC2L8h/4VpmGKURKl6UlVmjiMGpZ/y7/hdqfu165xn5hxcQYmpQqnYUeUkuETTWViVJtZGHmkroM7pjVY9Uaaztndk1YsNduNs4OmIV5wGgjTQKMQNqOU21bygzcxwFNOzM0tH1LoDhco+daSbGFdhtzjENvPKBF3234sobJFPfuuDgJ1GSY16+UrYYwVmvRaK1SQUffFQT6jGTnb7Az7n/6siugy18nULKI5dNB7XGDSnKSWT8WftnzuuQShukaUzZM9tH3BLU8aqc+3112jWP6XctDKOGed2QpemifcQFchzVj9srOUqsq4fVugrAWVqSU/ client@admin', u'kernel_id': u'', u'power_state': 1, u'default_ephemeral_device': u'/dev/vdb', u'progress': 0, u'project_id': u'c97027dd880d4c129ae7a4ba7edade05', u'launched_at': u'2012-11-05T20:43:22.000000', u'scheduled_at': u'2012-11-05T20:42:22.000000', u'ramdisk_id': u'', u'access_ip_v6': None, u'access_ip_v4': None, u'deleted': False, u'key_name': u'default', u'updated_at': u'2012-11-05T20:57:55.000000', u'host': u'cmp01', u'display_name': u'test', u'task_state': u'deleting', u'shutdown_terminate': False, u'architecture': None, u'root_gb': 10, u'locked': False, u'name': u'vm-0000014f', u'created_at': u'2012-11-05T20:42:22.000000', u'launch_index': 0, u'metadata': [], u'memory_mb': 2048, u'instance_type': {u'disabled': False, u'root_gb': 10, u'deleted_at': None, u'name': u'm1.small', u'deleted': False, u'created_at': None, u'ephemeral_gb': 20, u'updated_at': None, u'memory_mb': 2048, u'vcpus': 1, u'swap': 0, u'rxtx_factor': 1.0, u'is_public': True, u'flavorid': u'2', u'vcpu_weight': None, u'id': 5}, u'vcpus': 1, u'image_ref': u'0906f07d-f419-4a9d-8404-047db4717497', u'root_device_name': u'/dev/vda', u'auto_disk_config': None, u'os_type': None, u'config_drive': u''}}, u'_context_instance_lock_checked': False, u'_context_is_admin': False, u'version': u'2.0', u'_context_project_id': u'c97027dd880d4c129ae7a4ba7edade05', u'_context_timestamp': u'2012-11-05T21:01:48.753541', u'_context_read_deleted': u'no', u'_context_user_id': u'4ce5c085e09a478ea4edcd667a92df78', u'method': u'terminate_instance', u'_context_remote_address': u'192.168.5.51'} from (pid=33107) _safe_log /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/common.py:195
  2012-11-05 22:02:04 DEBUG nova.openstack.common.rpc.amqp [-] unpacked context: {'project_name': u'Smart', 'user_id': u'4ce5c085e09a478ea4edcd667a92df78', 'roles': [u'Member', u'admin'], 'timestamp': u'2012-11-05T21:01:48.753541', 'auth_token': '<SANITIZED>', 'remote_address': u'192.168.5.51', 'quota_class': None, 'is_admin': False, 'service_catalog': [{u'endpoints_links': [], u'endpoints': [{u'adminURL': u'http://nova.os.lan:8774/v2/c97027dd880d4c129ae7a4ba7edade05', u'region': u'RegionOne', u'publicURL': u'http://nova.bouvetlab.local:8774/v2/c97027dd880d4c129ae7a4ba7edade05', u'internalURL': u'http://nova.os.lan:8774/v2/c97027dd880d4c129ae7a4ba7edade05', u'id': u'7c348a9ac4744cedad5d13c6dd2617a5'}], u'type': u'compute', u'name': u'nova'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'http://quantum.os.lan:9696/', u'region': u'RegionOne', u'publicURL': u'http://quantum.bouvetlab.local:9696/', u'internalURL': u'http://quantum.os.lan:9696/', u'id': u'02bbf0add27040978d591f53c3ffedf0'}], u'type': u'network', u'name': u'quantum'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'http://glance.os.lan:9292/v2', u'region': u'RegionOne', u'publicURL': u'http://glance.bouvetlab.local:9292/v2', u'internalURL': u'http://glance.os.lan:9292/v2', u'id': u'c913e43a31744d499097234be00d5e6f'}], u'type': u'image', u'name': u'glance'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'http://192.168.6.12:8777', u'region': u'regionOne', u'publicURL': u'http://192.168.5.52:8777', u'internalURL': u'http://192.168.6.12:8777', u'id': u'c72653c1f8044a15874eb27a5fae580e'}], u'type': u'metering', u'name': u'ceilometer'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'http://heat.os.lan:8000/v1', u'region': u'RegionOne', u'publicURL': u'http://heat.os.lan:8000/v1', u'internalURL': u'http://heat.os.lan:8000/v1', u'id': u'bb5392c96c274710bdcd8646e2510c8b'}], u'type': u'cloudformation', u'name': u'heat'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'http://cinder.os.lan:8776/v1/c97027dd880d4c129ae7a4ba7edade05', u'region': u'RegionOne', u'publicURL': u'http://cinder.bouvetlab.local:8776/v1/c97027dd880d4c129ae7a4ba7edade05', u'internalURL': u'http://cinder.os.lan:8776/v1/c97027dd880d4c129ae7a4ba7edade05', u'id': u'e83311fd966c440384580df55cc5b5c4'}], u'type': u'volume', u'name': u'cinder'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'http://nova.os.lan:8773/services/Admin', u'region': u'RegionOne', u'publicURL': u'http://nova.bouvetlab.local:8773/services/Cloud', u'internalURL': u'http://nova.os.lan:8773/services/Cloud', u'id': u'46d5d9e4ec9d41cb8a2ea8b439cdd2ee'}], u'type': u'ec2', u'name': u'ec2'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'http://swift.os.lan:8080/v1', u'region': u'RegionOne', u'publicURL': u'http://swift.bouvetlab.local:8080/v1/AUTH_c97027dd880d4c129ae7a4ba7edade05', u'internalURL': u'http://swift.os.lan:8080/v1/AUTH_c97027dd880d4c129ae7a4ba7edade05', u'id': u'5f166002b1044286bc415b4fc89fdda8'}], u'type': u'object-store', u'name': u'swift'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'http://keystone.os.lan:35357/v2.0', u'region': u'RegionOne', u'publicURL': u'http://keystone.bouvetlab.local:5000/v2.0', u'internalURL': u'http://keystone.os.lan:5000/v2.0', u'id': u'788739bcbd06494785db65613ae2a289'}], u'type': u'identity', u'name': u'keystone'}], 'request_id': u'req-b71fb34b-6027-4725-88a0-9419d1e44d4e', 'instance_lock_checked': False, 'project_id': u'c97027dd880d4c129ae7a4ba7edade05', 'user_name': u'endre', 'read_deleted': u'no'} from (pid=33107) _safe_log /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/common.py:195
  2012-11-05 22:02:04 DEBUG nova.utils [req-b71fb34b-6027-4725-88a0-9419d1e44d4e 4ce5c085e09a478ea4edcd667a92df78 c97027dd880d4c129ae7a4ba7edade05] Got semaphore "8ea1adc7-b3af-4892-8fa8-3c8e76f5dd68" for method "do_terminate_instance"... from (pid=33107) inner /usr/lib/python2.7/dist-packages/nova/utils.py:713
  2012-11-05 22:02:04 DEBUG nova.openstack.common.rpc.amqp [-] Sending compute.instance.delete.start on notifications.info from (pid=33107) notify /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py:410
  2012-11-05 22:02:04 DEBUG nova.openstack.common.rpc.amqp [-] Pool creating new connection from (pid=33107) create /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py:57
  2012-11-05 22:02:04 INFO nova.openstack.common.rpc.common [-] Connected to AMQP server on nova.os.lan:5672
  2012-11-05 22:02:04 DEBUG nova.ceilometer.notifier [-] polling final stats for u'8ea1adc7-b3af-4892-8fa8-3c8e76f5dd68' from (pid=33107) notify /opt/ceilometer/ceilometer/compute/nova_notifier.py:62
  2012-11-05 22:02:04 DEBUG nova.virt.libvirt.driver [req-b71fb34b-6027-4725-88a0-9419d1e44d4e 4ce5c085e09a478ea4edcd667a92df78 c97027dd880d4c129ae7a4ba7edade05] Connecting to libvirt: qemu:///system from (pid=33107) _get_connection /usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py:340
    2012-11-05 22:02:48 DEBUG nova.manager [req-9f2ef547-c3a5-4be2-b4df-de9f402c7232 None None] Running periodic task ComputeManager._publish_service_capabilities from (pid=33107) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:172
  2012-11-05 22:02:48 DEBUG nova.manager [req-9f2ef547-c3a5-4be2-b4df-de9f402c7232 None None] Running periodic task ComputeManager._poll_rescued_instances from (pid=33107) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:172
  2012-11-05 22:02:48 DEBUG nova.manager [req-9f2ef547-c3a5-4be2-b4df-de9f402c7232 None None] Skipping ComputeManager._sync_power_states, 10 ticks left until next run from (pid=33107) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:167
  2012-11-05 22:02:48 DEBUG nova.manager [req-9f2ef547-c3a5-4be2-b4df-de9f402c7232 None None] Running periodic task ComputeManager._poll_bandwidth_usage from (pid=33107) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:172

To manage notifications about this bug go to:
https://bugs.launchpad.net/ceilometer/+bug/1075320/+subscriptions
  Thread Previous • Date Previous • • Thread Next