← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 1186243] Re: Race can cause soft deleted instance to be erased by mistake

 

** Changed in: nova/grizzly
       Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1186243

Title:
  Race can cause soft deleted instance to be erased by mistake

Status in OpenStack Compute (Nova):
  Fix Released
Status in OpenStack Compute (nova) grizzly series:
  Fix Released

Bug description:
  Got this error from gerrit.

  python26 passed, but python27 failed.

  ======================================================================
  2013-05-31 12:11:53.737 | FAIL: nova.tests.integrated.test_servers.ServersTest.test_deferred_delete_restore
  2013-05-31 12:11:53.737 | tags: worker-3
  2013-05-31 12:11:53.737 | ----------------------------------------------------------------------
  2013-05-31 12:11:53.737 | Empty attachments:
  2013-05-31 12:11:53.737 |   stderr
  2013-05-31 12:11:53.737 |   stdout
  2013-05-31 12:11:53.737 | 
  2013-05-31 12:11:53.737 | pythonlogging:'': {{{
  2013-05-31 12:11:53.738 | Starting conductor node (version 2013.2)
  2013-05-31 12:11:53.738 | Loading compute driver 'nova.virt.fake.FakeDriver'
  2013-05-31 12:11:53.738 | Starting compute node (version 2013.2)
  2013-05-31 12:11:53.738 | Updating host status
  2013-05-31 12:11:53.738 | Auditing locally available compute resources
  2013-05-31 12:11:53.738 | Free ram (MB): 7680
  2013-05-31 12:11:53.738 | Free disk (GB): 1028
  2013-05-31 12:11:53.738 | Free VCPUS: 1
  2013-05-31 12:11:53.738 | Compute_service record created for 760497b8a7e84358b92c6bbe81556192:fake-mini
  2013-05-31 12:11:53.738 | Starting cert node (version 2013.2)
  2013-05-31 12:11:53.738 | Loading network driver 'nova.network.linux_net'
  2013-05-31 12:11:53.738 | Starting network node (version 2013.2)
  2013-05-31 12:11:53.738 | Starting scheduler node (version 2013.2)
  2013-05-31 12:11:53.738 | Starting cells node (version 2013.2)
  2013-05-31 12:11:53.738 | Initializing extension manager.
  2013-05-31 12:11:53.738 | Loaded extension: os-fping
  2013-05-31 12:11:53.739 | Loaded extension: os-services
  2013-05-31 12:11:53.739 | Loaded extension: os-hosts
  2013-05-31 12:11:53.739 | Loaded extension: os-create-server-ext
  2013-05-31 12:11:53.739 | Loaded extension: OS-EXT-SRV-ATTR
  2013-05-31 12:11:53.739 | Loaded extension: OS-EXT-IPS-MAC
  2013-05-31 12:11:53.739 | Loaded extension: OS-SCH-HNT
  2013-05-31 12:11:53.739 | Loaded extension: os-floating-ips-bulk
  2013-05-31 12:11:53.739 | Loaded extension: os-keypairs
  2013-05-31 12:11:53.739 | Loaded extension: OS-EXT-VIF-NET
  2013-05-31 12:11:53.739 | Loaded extension: os-extended-floating-ips
  2013-05-31 12:11:53.739 | Loaded extension: os-user-data
  2013-05-31 12:11:53.739 | Loaded extension: os-virtual-interfaces
  2013-05-31 12:11:53.739 | Loaded extension: os-hide-server-addresses
  2013-05-31 12:11:53.739 | Loaded extension: os-agents
  2013-05-31 12:11:53.739 | Loaded extension: os-quota-sets
  2013-05-31 12:11:53.740 | Loaded extension: os-deferred-delete
  2013-05-31 12:11:53.740 | Loaded extension: os-cells
  2013-05-31 12:11:53.740 | Loaded extension: os-tenant-networks
  2013-05-31 12:11:53.740 | Loaded extension: os-hypervisors
  2013-05-31 12:11:53.740 | Loaded extension: os-quota-class-sets
  2013-05-31 12:11:53.740 | Loaded extension: os-server-password
  2013-05-31 12:11:53.740 | Loaded extension: OS-EXT-AZ
  2013-05-31 12:11:53.740 | Loaded extension: os-rescue
  2013-05-31 12:11:53.740 | Loaded extension: OS-EXT-IPS
  2013-05-31 12:11:53.740 | Loaded extension: OS-EXT-STS
  2013-05-31 12:11:53.740 | Loaded extension: os-security-group-default-rules
  2013-05-31 12:11:53.740 | Loaded extension: os-certificates
  2013-05-31 12:11:53.740 | Loaded extension: os-availability-zone
  2013-05-31 12:11:53.740 | Loaded extension: os-networks
  2013-05-31 12:11:53.740 | Loaded extension: os-flavor-swap
  2013-05-31 12:11:53.741 | Loaded extension: os-instance-actions
  2013-05-31 12:11:53.741 | Loaded extension: os-flavor-manage
  2013-05-31 12:11:53.741 | Loaded extension: os-flavor-extra-specs
  2013-05-31 12:11:53.741 | Loaded extension: os-floating-ip-pools
  2013-05-31 12:11:53.741 | Loaded extension: OS-FLV-DISABLED
  2013-05-31 12:11:53.741 | Loaded extension: os-cloudpipe
  2013-05-31 12:11:53.741 | Loaded extension: os-config-drive
  2013-05-31 12:11:53.741 | Loaded extension: os-floating-ip-dns
  2013-05-31 12:11:53.741 | Loaded extension: os-volumes
  2013-05-31 12:11:53.741 | Loaded extension: os-multiple-create
  2013-05-31 12:11:53.741 | Loaded extension: os-flavor-rxtx
  2013-05-31 12:11:53.741 | Loaded extension: os-networks-associate
  2013-05-31 12:11:53.741 | Loaded extension: os-simple-tenant-usage
  2013-05-31 12:11:53.741 | Loaded extension: NMN
  2013-05-31 12:11:53.741 | Loaded extension: os-floating-ips
  2013-05-31 12:11:53.742 | Loaded extension: os-extended-quotas
  2013-05-31 12:11:53.742 | Loaded extension: os-coverage
  2013-05-31 12:11:53.742 | Loaded extension: os-aggregates
  2013-05-31 12:11:53.742 | Loaded extension: os-server-diagnostics
  2013-05-31 12:11:53.742 | Loaded extension: os-console-output
  2013-05-31 12:11:53.742 | Loaded extension: os-security-groups
  2013-05-31 12:11:53.742 | Loaded extension: os-evacuate
  2013-05-31 12:11:53.742 | Loaded extension: os-used-limits
  2013-05-31 12:11:53.742 | Loaded extension: OS-FLV-EXT-DATA
  2013-05-31 12:11:53.742 | Loaded extension: OS-EXT-IMG-SIZE
  2013-05-31 12:11:53.742 | Loaded extension: os-server-start-stop
  2013-05-31 12:11:53.742 | Loaded extension: os-cloudpipe-update
  2013-05-31 12:11:53.742 | Loaded extension: OS-DCF
  2013-05-31 12:11:53.742 | Loaded extension: os-admin-actions
  2013-05-31 12:11:53.743 | Loaded extension: os-attach-interfaces
  2013-05-31 12:11:53.743 | Loaded extension: os-fixed-ips
  2013-05-31 12:11:53.743 | Loaded extension: os-instance_usage_audit_log
  2013-05-31 12:11:53.743 | Loaded extension: os-consoles
  2013-05-31 12:11:53.743 | Loaded extension: os-flavor-access
  2013-05-31 12:11:53.743 | Loaded extension: os-baremetal-nodes
  2013-05-31 12:11:53.743 | Loaded extension os-keypairs
  2013-05-31 12:11:53.743 | Loaded extension os-fixed-ips
  2013-05-31 12:11:53.743 | Loaded extension extensions
  2013-05-31 12:11:53.743 | Loaded extension servers
  2013-05-31 12:11:53.743 | Initializing extension manager.
  2013-05-31 12:11:53.743 | Loaded extension: os-fping
  2013-05-31 12:11:53.743 | Loaded extension: os-services
  2013-05-31 12:11:53.744 | Loaded extension: os-hosts
  2013-05-31 12:11:53.744 | Loaded extension: os-create-server-ext
  2013-05-31 12:11:53.744 | Loaded extension: OS-EXT-SRV-ATTR
  2013-05-31 12:11:53.744 | Loaded extension: OS-EXT-IPS-MAC
  2013-05-31 12:11:53.744 | Loaded extension: OS-SCH-HNT
  2013-05-31 12:11:53.744 | Loaded extension: os-floating-ips-bulk
  2013-05-31 12:11:53.744 | Loaded extension: os-keypairs
  2013-05-31 12:11:53.744 | Loaded extension: OS-EXT-VIF-NET
  2013-05-31 12:11:53.744 | Loaded extension: os-extended-floating-ips
  2013-05-31 12:11:53.744 | Loaded extension: os-user-data
  2013-05-31 12:11:53.744 | Loaded extension: os-virtual-interfaces
  2013-05-31 12:11:53.744 | Loaded extension: os-hide-server-addresses
  2013-05-31 12:11:53.744 | Loaded extension: os-agents
  2013-05-31 12:11:53.744 | Loaded extension: os-quota-sets
  2013-05-31 12:11:53.744 | Loaded extension: os-deferred-delete
  2013-05-31 12:11:53.744 | Loaded extension: os-cells
  2013-05-31 12:11:53.745 | Loaded extension: os-tenant-networks
  2013-05-31 12:11:53.745 | Loaded extension: os-hypervisors
  2013-05-31 12:11:53.745 | Loaded extension: os-quota-class-sets
  2013-05-31 12:11:53.745 | Loaded extension: os-server-password
  2013-05-31 12:11:53.745 | Loaded extension: OS-EXT-AZ
  2013-05-31 12:11:53.745 | Loaded extension: os-rescue
  2013-05-31 12:11:53.745 | Loaded extension: OS-EXT-IPS
  2013-05-31 12:11:53.745 | Loaded extension: OS-EXT-STS
  2013-05-31 12:11:53.745 | Loaded extension: os-security-group-default-rules
  2013-05-31 12:11:53.745 | Loaded extension: os-certificates
  2013-05-31 12:11:53.745 | Loaded extension: os-availability-zone
  2013-05-31 12:11:53.745 | Loaded extension: os-networks
  2013-05-31 12:11:53.745 | Loaded extension: os-flavor-swap
  2013-05-31 12:11:53.745 | Loaded extension: os-instance-actions
  2013-05-31 12:11:53.746 | Loaded extension: os-flavor-manage
  2013-05-31 12:11:53.746 | Loaded extension: os-flavor-extra-specs
  2013-05-31 12:11:53.746 | Loaded extension: os-floating-ip-pools
  2013-05-31 12:11:53.746 | Loaded extension: OS-FLV-DISABLED
  2013-05-31 12:11:53.746 | Loaded extension: os-cloudpipe
  2013-05-31 12:11:53.746 | Loaded extension: os-config-drive
  2013-05-31 12:11:53.746 | Loaded extension: os-floating-ip-dns
  2013-05-31 12:11:53.746 | Loaded extension: os-volumes
  2013-05-31 12:11:53.746 | Loaded extension: os-multiple-create
  2013-05-31 12:11:53.746 | Loaded extension: os-flavor-rxtx
  2013-05-31 12:11:53.746 | Loaded extension: os-networks-associate
  2013-05-31 12:11:53.746 | Loaded extension: os-simple-tenant-usage
  2013-05-31 12:11:53.746 | Loaded extension: NMN
  2013-05-31 12:11:53.746 | Loaded extension: os-floating-ips
  2013-05-31 12:11:53.746 | Loaded extension: os-extended-quotas
  2013-05-31 12:11:53.746 | Loaded extension: os-coverage
  2013-05-31 12:11:53.747 | Loaded extension: os-aggregates
  2013-05-31 12:11:53.747 | Loaded extension: os-server-diagnostics
  2013-05-31 12:11:53.747 | Loaded extension: os-console-output
  2013-05-31 12:11:53.747 | Loaded extension: os-security-groups
  2013-05-31 12:11:53.747 | Loaded extension: os-evacuate
  2013-05-31 12:11:53.747 | Loaded extension: os-used-limits
  2013-05-31 12:11:53.747 | Loaded extension: OS-FLV-EXT-DATA
  2013-05-31 12:11:53.747 | Loaded extension: OS-EXT-IMG-SIZE
  2013-05-31 12:11:53.747 | Loaded extension: os-server-start-stop
  2013-05-31 12:11:53.747 | Loaded extension: os-cloudpipe-update
  2013-05-31 12:11:53.747 | Loaded extension: OS-DCF
  2013-05-31 12:11:53.747 | Loaded extension: os-admin-actions
  2013-05-31 12:11:53.747 | Loaded extension: os-attach-interfaces
  2013-05-31 12:11:53.747 | Loaded extension: os-fixed-ips
  2013-05-31 12:11:53.747 | Loaded extension: os-instance_usage_audit_log
  2013-05-31 12:11:53.748 | Loaded extension: os-consoles
  2013-05-31 12:11:53.748 | Loaded extension: os-flavor-access
  2013-05-31 12:11:53.748 | Loaded extension: os-baremetal-nodes
  2013-05-31 12:11:53.748 | osapi_compute listening on 127.0.0.1:40408
  2013-05-31 12:11:53.748 | http://127.0.0.1:40408/v2
  2013-05-31 12:11:53.748 | Starting conductor node (version 2013.2)
  2013-05-31 12:11:53.748 | (31622) wsgi starting up on http://127.0.0.1:40408/
  2013-05-31 12:11:53.748 | 
  2013-05-31 12:11:53.748 | Doing GET on /v2
  2013-05-31 12:11:53.748 | (31622) accepted ('127.0.0.1', 60809)
  2013-05-31 12:11:53.748 | 
  2013-05-31 12:11:53.748 | 127.0.0.1 "GET /v2 HTTP/1.1" status: 204 len: 216 time: 0.0006292
  2013-05-31 12:11:53.748 | 
  2013-05-31 12:11:53.748 | Doing GET on /v2/openstack//images/detail
  2013-05-31 12:11:53.748 | (31622) accepted ('127.0.0.1', 60810)
  2013-05-31 12:11:53.748 | 
  2013-05-31 12:11:53.749 | 127.0.0.1 "GET /v2/openstack//images/detail HTTP/1.1" status: 200 len: 5251 time: 0.8105230
  2013-05-31 12:11:53.749 | 
  2013-05-31 12:11:53.749 | Doing GET on /v2/openstack//flavors/detail
  2013-05-31 12:11:53.749 | (31622) accepted ('127.0.0.1', 60813)
  2013-05-31 12:11:53.749 | 
  2013-05-31 12:11:53.749 | 127.0.0.1 "GET /v2/openstack//flavors/detail HTTP/1.1" status: 200 len: 2013 time: 0.0107400
  2013-05-31 12:11:53.749 | 
  2013-05-31 12:11:53.749 | Doing GET on /v2/openstack//servers/detail
  2013-05-31 12:11:53.749 | (31622) accepted ('127.0.0.1', 60814)
  2013-05-31 12:11:53.749 | 
  2013-05-31 12:11:53.749 | 127.0.0.1 "GET /v2/openstack//servers/detail HTTP/1.1" status: 200 len: 187 time: 0.0314510
  2013-05-31 12:11:53.749 | 
  2013-05-31 12:11:53.749 | Doing POST on /v2/openstack//servers
  2013-05-31 12:11:53.749 | Body: {"server": {"flavorRef": "http://fake.server/1";, "name": "serverGNYOHCDP", "imageRef": "http://fake.server/70a599e0-31e7-49b7-b260-868f441e862b"}}
  2013-05-31 12:11:53.749 | (31622) accepted ('127.0.0.1', 60815)
  2013-05-31 12:11:53.750 | 
  2013-05-31 12:11:53.750 | Starting instance...
  2013-05-31 12:11:53.750 | Attempting claim: memory 512 MB, disk 1 GB, VCPUs 1
  2013-05-31 12:11:53.750 | Total Memory: 8192 MB, used: 512.00 MB
  2013-05-31 12:11:53.750 | Memory limit not specified, defaulting to unlimited
  2013-05-31 12:11:53.750 | Total Disk: 1028 GB, used: 0.00 GB
  2013-05-31 12:11:53.750 | Disk limit not specified, defaulting to unlimited
  2013-05-31 12:11:53.750 | Total CPU: 1 VCPUs, used: 0.00 VCPUs
  2013-05-31 12:11:53.750 | CPU limit not specified, defaulting to unlimited
  2013-05-31 12:11:53.750 | Claim successful
  2013-05-31 12:11:53.750 | 127.0.0.1 "POST /v2/openstack//servers HTTP/1.1" status: 202 len: 659 time: 0.5113389
  2013-05-31 12:13:26.360 | 
  2013-05-31 12:13:26.360 | Doing GET on /v2/openstack//servers/42e6f3a9-6618-4197-97d4-67c7dbe33bfd
  2013-05-31 12:13:26.360 | (31622) accepted ('127.0.0.1', 60817)
  2013-05-31 12:13:26.360 | 
  2013-05-31 12:13:26.360 | 127.0.0.1 "GET /v2/openstack//servers/42e6f3a9-6618-4197-97d4-67c7dbe33bfd HTTP/1.1" status: 200 len: 1470 time: 0.0540881
  2013-05-31 12:13:26.360 | 
  2013-05-31 12:13:26.360 | Doing DELETE on /v2/openstack//servers/42e6f3a9-6618-4197-97d4-67c7dbe33bfd
  2013-05-31 12:13:26.361 | (31622) accepted ('127.0.0.1', 60818)
  2013-05-31 12:13:26.361 | 
  2013-05-31 12:13:26.361 | Auditing locally available compute resources
  2013-05-31 12:13:26.361 | 127.0.0.1 "DELETE /v2/openstack//servers/42e6f3a9-6618-4197-97d4-67c7dbe33bfd HTTP/1.1" status: 204 len: 179 time: 0.2205250
  2013-05-31 12:13:26.361 | 
  2013-05-31 12:13:26.361 | Doing GET on /v2/openstack//servers/42e6f3a9-6618-4197-97d4-67c7dbe33bfd
  2013-05-31 12:13:26.361 | (31622) accepted ('127.0.0.1', 60821)
  2013-05-31 12:13:26.361 | 
  2013-05-31 12:13:26.361 | 127.0.0.1 "GET /v2/openstack//servers/42e6f3a9-6618-4197-97d4-67c7dbe33bfd HTTP/1.1" status: 200 len: 1461 time: 0.0791640
  2013-05-31 12:13:26.361 | 
  2013-05-31 12:13:26.361 | Doing POST on /v2/openstack//servers/42e6f3a9-6618-4197-97d4-67c7dbe33bfd/action
  2013-05-31 12:13:26.361 | Body: {"restore": {}}
  2013-05-31 12:13:26.361 | Free ram (MB): 7168
  2013-05-31 12:13:26.361 | Free disk (GB): 1027
  2013-05-31 12:13:26.362 | Free VCPUS: 0
  2013-05-31 12:13:26.362 | (31622) accepted ('127.0.0.1', 60822)
  2013-05-31 12:13:26.362 | 
  2013-05-31 12:13:26.362 | Compute_service record updated for 760497b8a7e84358b92c6bbe81556192:fake-mini
  2013-05-31 12:13:26.362 | Reclaiming deleted instance
  2013-05-31 12:13:26.362 | Terminating instance
  2013-05-31 12:13:26.362 | 127.0.0.1 "POST /v2/openstack//servers/42e6f3a9-6618-4197-97d4-67c7dbe33bfd/action HTTP/1.1" status: 202 len: 121 time: 0.2906492
  2013-05-31 12:13:26.362 | 
  2013-05-31 12:13:26.362 | Doing GET on /v2/openstack//servers/42e6f3a9-6618-4197-97d4-67c7dbe33bfd
  2013-05-31 12:13:26.362 | (31622) accepted ('127.0.0.1', 60823)
  2013-05-31 12:13:26.362 | 
  2013-05-31 12:13:26.362 | 127.0.0.1 "GET /v2/openstack//servers/42e6f3a9-6618-4197-97d4-67c7dbe33bfd HTTP/1.1" status: 200 len: 1457 time: 0.0971241
  2013-05-31 12:13:26.362 | 
  2013-05-31 12:13:26.362 | Updating bandwidth usage cache
  2013-05-31 12:13:26.362 | Doing GET on /v2/openstack//servers/42e6f3a9-6618-4197-97d4-67c7dbe33bfd
  2013-05-31 12:13:26.362 | (31622) accepted ('127.0.0.1', 60825)
  2013-05-31 12:13:26.363 | 
  2013-05-31 12:13:26.363 | HTTP exception thrown: Instance could not be found
  2013-05-31 12:13:26.363 | 127.0.0.1 "GET /v2/openstack//servers/42e6f3a9-6618-4197-97d4-67c7dbe33bfd HTTP/1.1" status: 404 len: 267 time: 0.0434902
  2013-05-31 12:13:26.363 | 
  2013-05-31 12:13:26.363 | Stopping WSGI server.
  2013-05-31 12:13:26.363 | }}}
  2013-05-31 12:13:26.363 | 
  2013-05-31 12:13:26.363 | Traceback (most recent call last):
  2013-05-31 12:13:26.363 |   File "/home/jenkins/workspace/gate-nova-python27/nova/tests/integrated/test_servers.py", line 216, in test_deferred_delete_restore
  2013-05-31 12:13:26.363 |     found_server = self._wait_for_state_change(found_server, 'DELETED')
  2013-05-31 12:13:26.363 |   File "/home/jenkins/workspace/gate-nova-python27/nova/tests/integrated/test_servers.py", line 39, in _wait_for_state_change
  2013-05-31 12:13:26.363 |     server = self.api.get_server(server['id'])
  2013-05-31 12:13:26.363 |   File "/home/jenkins/workspace/gate-nova-python27/nova/tests/integrated/api/client.py", line 211, in get_server
  2013-05-31 12:13:26.363 |     return self.api_get('/servers/%s' % server_id)['server']
  2013-05-31 12:13:26.363 |   File "/home/jenkins/workspace/gate-nova-python27/nova/tests/integrated/api/client.py", line 180, in api_get
  2013-05-31 12:13:26.364 |     response = self.api_request(relative_uri, **kwargs)
  2013-05-31 12:13:26.364 |   File "/home/jenkins/workspace/gate-nova-python27/nova/tests/integrated/api/client.py", line 160, in api_request
  2013-05-31 12:13:26.364 |     raise OpenStackApiNotFoundException(response=response)
  2013-05-31 12:13:26.364 | OpenStackApiNotFoundException: Item not found
  2013-05-31 12:13:26.364 | Status Code: 404
  2013-05-31 12:13:26.364 | Body: {"itemNotFound": {"message": "Instance could not be found", "code": 404}}
  2013-05-31 12:13:26.364 | ======================================================================
  2013-05-31 12:13:26.364 | FAIL: process-returncode
  2013-05-31 12:13:26.364 | tags: worker-3
  2013-05-31 12:13:26.364 | ----------------------------------------------------------------------
  2013-05-31 12:13:26.364 | Binary content:
  2013-05-31 12:13:26.364 |   traceback (test/plain; charset="utf8")
  2013-05-31 12:13:26.364 | Ran 6154 tests in 322.356s
  2013-05-31 12:13:26.364 | FAILED (id=0, failures=2, skips=8)
  2013-05-31 12:13:26.364 | error: testr failed (1)
  2013-05-31 12:13:26.581 | ERROR: InvocationError: '/home/jenkins/workspace/gate-nova-python27/.tox/py27/bin/python setup.py testr --slowest --testr-args='
  2013-05-31 12:13:26.582 | ___________________________________ summary ____________________________________
  2013-05-31 12:13:26.582 | ERROR:   py27: commands failed
  2013-05-31 12:13:26.600 | + result=1
  2013-05-31 12:13:26.600 | + echo 'Begin pip freeze output from test virtualenv:'
  2013-05-31 12:13:26.600 | Begin pip freeze output from test virtualenv:
  2013-05-31 12:13:26.600 | + echo ======================================================================
  2013-05-31 12:13:26.600 | ======================================================================

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1186243/+subscriptions