yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #03870
[Bug 1186243] Re: Race can cause soft deleted instance to be erased by mistake
** Changed in: nova
Status: Fix Committed => Fix Released
** Changed in: nova
Milestone: None => havana-2
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1186243
Title:
Race can cause soft deleted instance to be erased by mistake
Status in OpenStack Compute (Nova):
Fix Released
Status in OpenStack Compute (nova) grizzly series:
In Progress
Bug description:
Got this error from gerrit.
python26 passed, but python27 failed.
======================================================================
2013-05-31 12:11:53.737 | FAIL: nova.tests.integrated.test_servers.ServersTest.test_deferred_delete_restore
2013-05-31 12:11:53.737 | tags: worker-3
2013-05-31 12:11:53.737 | ----------------------------------------------------------------------
2013-05-31 12:11:53.737 | Empty attachments:
2013-05-31 12:11:53.737 | stderr
2013-05-31 12:11:53.737 | stdout
2013-05-31 12:11:53.737 |
2013-05-31 12:11:53.737 | pythonlogging:'': {{{
2013-05-31 12:11:53.738 | Starting conductor node (version 2013.2)
2013-05-31 12:11:53.738 | Loading compute driver 'nova.virt.fake.FakeDriver'
2013-05-31 12:11:53.738 | Starting compute node (version 2013.2)
2013-05-31 12:11:53.738 | Updating host status
2013-05-31 12:11:53.738 | Auditing locally available compute resources
2013-05-31 12:11:53.738 | Free ram (MB): 7680
2013-05-31 12:11:53.738 | Free disk (GB): 1028
2013-05-31 12:11:53.738 | Free VCPUS: 1
2013-05-31 12:11:53.738 | Compute_service record created for 760497b8a7e84358b92c6bbe81556192:fake-mini
2013-05-31 12:11:53.738 | Starting cert node (version 2013.2)
2013-05-31 12:11:53.738 | Loading network driver 'nova.network.linux_net'
2013-05-31 12:11:53.738 | Starting network node (version 2013.2)
2013-05-31 12:11:53.738 | Starting scheduler node (version 2013.2)
2013-05-31 12:11:53.738 | Starting cells node (version 2013.2)
2013-05-31 12:11:53.738 | Initializing extension manager.
2013-05-31 12:11:53.738 | Loaded extension: os-fping
2013-05-31 12:11:53.739 | Loaded extension: os-services
2013-05-31 12:11:53.739 | Loaded extension: os-hosts
2013-05-31 12:11:53.739 | Loaded extension: os-create-server-ext
2013-05-31 12:11:53.739 | Loaded extension: OS-EXT-SRV-ATTR
2013-05-31 12:11:53.739 | Loaded extension: OS-EXT-IPS-MAC
2013-05-31 12:11:53.739 | Loaded extension: OS-SCH-HNT
2013-05-31 12:11:53.739 | Loaded extension: os-floating-ips-bulk
2013-05-31 12:11:53.739 | Loaded extension: os-keypairs
2013-05-31 12:11:53.739 | Loaded extension: OS-EXT-VIF-NET
2013-05-31 12:11:53.739 | Loaded extension: os-extended-floating-ips
2013-05-31 12:11:53.739 | Loaded extension: os-user-data
2013-05-31 12:11:53.739 | Loaded extension: os-virtual-interfaces
2013-05-31 12:11:53.739 | Loaded extension: os-hide-server-addresses
2013-05-31 12:11:53.739 | Loaded extension: os-agents
2013-05-31 12:11:53.739 | Loaded extension: os-quota-sets
2013-05-31 12:11:53.740 | Loaded extension: os-deferred-delete
2013-05-31 12:11:53.740 | Loaded extension: os-cells
2013-05-31 12:11:53.740 | Loaded extension: os-tenant-networks
2013-05-31 12:11:53.740 | Loaded extension: os-hypervisors
2013-05-31 12:11:53.740 | Loaded extension: os-quota-class-sets
2013-05-31 12:11:53.740 | Loaded extension: os-server-password
2013-05-31 12:11:53.740 | Loaded extension: OS-EXT-AZ
2013-05-31 12:11:53.740 | Loaded extension: os-rescue
2013-05-31 12:11:53.740 | Loaded extension: OS-EXT-IPS
2013-05-31 12:11:53.740 | Loaded extension: OS-EXT-STS
2013-05-31 12:11:53.740 | Loaded extension: os-security-group-default-rules
2013-05-31 12:11:53.740 | Loaded extension: os-certificates
2013-05-31 12:11:53.740 | Loaded extension: os-availability-zone
2013-05-31 12:11:53.740 | Loaded extension: os-networks
2013-05-31 12:11:53.740 | Loaded extension: os-flavor-swap
2013-05-31 12:11:53.741 | Loaded extension: os-instance-actions
2013-05-31 12:11:53.741 | Loaded extension: os-flavor-manage
2013-05-31 12:11:53.741 | Loaded extension: os-flavor-extra-specs
2013-05-31 12:11:53.741 | Loaded extension: os-floating-ip-pools
2013-05-31 12:11:53.741 | Loaded extension: OS-FLV-DISABLED
2013-05-31 12:11:53.741 | Loaded extension: os-cloudpipe
2013-05-31 12:11:53.741 | Loaded extension: os-config-drive
2013-05-31 12:11:53.741 | Loaded extension: os-floating-ip-dns
2013-05-31 12:11:53.741 | Loaded extension: os-volumes
2013-05-31 12:11:53.741 | Loaded extension: os-multiple-create
2013-05-31 12:11:53.741 | Loaded extension: os-flavor-rxtx
2013-05-31 12:11:53.741 | Loaded extension: os-networks-associate
2013-05-31 12:11:53.741 | Loaded extension: os-simple-tenant-usage
2013-05-31 12:11:53.741 | Loaded extension: NMN
2013-05-31 12:11:53.741 | Loaded extension: os-floating-ips
2013-05-31 12:11:53.742 | Loaded extension: os-extended-quotas
2013-05-31 12:11:53.742 | Loaded extension: os-coverage
2013-05-31 12:11:53.742 | Loaded extension: os-aggregates
2013-05-31 12:11:53.742 | Loaded extension: os-server-diagnostics
2013-05-31 12:11:53.742 | Loaded extension: os-console-output
2013-05-31 12:11:53.742 | Loaded extension: os-security-groups
2013-05-31 12:11:53.742 | Loaded extension: os-evacuate
2013-05-31 12:11:53.742 | Loaded extension: os-used-limits
2013-05-31 12:11:53.742 | Loaded extension: OS-FLV-EXT-DATA
2013-05-31 12:11:53.742 | Loaded extension: OS-EXT-IMG-SIZE
2013-05-31 12:11:53.742 | Loaded extension: os-server-start-stop
2013-05-31 12:11:53.742 | Loaded extension: os-cloudpipe-update
2013-05-31 12:11:53.742 | Loaded extension: OS-DCF
2013-05-31 12:11:53.742 | Loaded extension: os-admin-actions
2013-05-31 12:11:53.743 | Loaded extension: os-attach-interfaces
2013-05-31 12:11:53.743 | Loaded extension: os-fixed-ips
2013-05-31 12:11:53.743 | Loaded extension: os-instance_usage_audit_log
2013-05-31 12:11:53.743 | Loaded extension: os-consoles
2013-05-31 12:11:53.743 | Loaded extension: os-flavor-access
2013-05-31 12:11:53.743 | Loaded extension: os-baremetal-nodes
2013-05-31 12:11:53.743 | Loaded extension os-keypairs
2013-05-31 12:11:53.743 | Loaded extension os-fixed-ips
2013-05-31 12:11:53.743 | Loaded extension extensions
2013-05-31 12:11:53.743 | Loaded extension servers
2013-05-31 12:11:53.743 | Initializing extension manager.
2013-05-31 12:11:53.743 | Loaded extension: os-fping
2013-05-31 12:11:53.743 | Loaded extension: os-services
2013-05-31 12:11:53.744 | Loaded extension: os-hosts
2013-05-31 12:11:53.744 | Loaded extension: os-create-server-ext
2013-05-31 12:11:53.744 | Loaded extension: OS-EXT-SRV-ATTR
2013-05-31 12:11:53.744 | Loaded extension: OS-EXT-IPS-MAC
2013-05-31 12:11:53.744 | Loaded extension: OS-SCH-HNT
2013-05-31 12:11:53.744 | Loaded extension: os-floating-ips-bulk
2013-05-31 12:11:53.744 | Loaded extension: os-keypairs
2013-05-31 12:11:53.744 | Loaded extension: OS-EXT-VIF-NET
2013-05-31 12:11:53.744 | Loaded extension: os-extended-floating-ips
2013-05-31 12:11:53.744 | Loaded extension: os-user-data
2013-05-31 12:11:53.744 | Loaded extension: os-virtual-interfaces
2013-05-31 12:11:53.744 | Loaded extension: os-hide-server-addresses
2013-05-31 12:11:53.744 | Loaded extension: os-agents
2013-05-31 12:11:53.744 | Loaded extension: os-quota-sets
2013-05-31 12:11:53.744 | Loaded extension: os-deferred-delete
2013-05-31 12:11:53.744 | Loaded extension: os-cells
2013-05-31 12:11:53.745 | Loaded extension: os-tenant-networks
2013-05-31 12:11:53.745 | Loaded extension: os-hypervisors
2013-05-31 12:11:53.745 | Loaded extension: os-quota-class-sets
2013-05-31 12:11:53.745 | Loaded extension: os-server-password
2013-05-31 12:11:53.745 | Loaded extension: OS-EXT-AZ
2013-05-31 12:11:53.745 | Loaded extension: os-rescue
2013-05-31 12:11:53.745 | Loaded extension: OS-EXT-IPS
2013-05-31 12:11:53.745 | Loaded extension: OS-EXT-STS
2013-05-31 12:11:53.745 | Loaded extension: os-security-group-default-rules
2013-05-31 12:11:53.745 | Loaded extension: os-certificates
2013-05-31 12:11:53.745 | Loaded extension: os-availability-zone
2013-05-31 12:11:53.745 | Loaded extension: os-networks
2013-05-31 12:11:53.745 | Loaded extension: os-flavor-swap
2013-05-31 12:11:53.745 | Loaded extension: os-instance-actions
2013-05-31 12:11:53.746 | Loaded extension: os-flavor-manage
2013-05-31 12:11:53.746 | Loaded extension: os-flavor-extra-specs
2013-05-31 12:11:53.746 | Loaded extension: os-floating-ip-pools
2013-05-31 12:11:53.746 | Loaded extension: OS-FLV-DISABLED
2013-05-31 12:11:53.746 | Loaded extension: os-cloudpipe
2013-05-31 12:11:53.746 | Loaded extension: os-config-drive
2013-05-31 12:11:53.746 | Loaded extension: os-floating-ip-dns
2013-05-31 12:11:53.746 | Loaded extension: os-volumes
2013-05-31 12:11:53.746 | Loaded extension: os-multiple-create
2013-05-31 12:11:53.746 | Loaded extension: os-flavor-rxtx
2013-05-31 12:11:53.746 | Loaded extension: os-networks-associate
2013-05-31 12:11:53.746 | Loaded extension: os-simple-tenant-usage
2013-05-31 12:11:53.746 | Loaded extension: NMN
2013-05-31 12:11:53.746 | Loaded extension: os-floating-ips
2013-05-31 12:11:53.746 | Loaded extension: os-extended-quotas
2013-05-31 12:11:53.746 | Loaded extension: os-coverage
2013-05-31 12:11:53.747 | Loaded extension: os-aggregates
2013-05-31 12:11:53.747 | Loaded extension: os-server-diagnostics
2013-05-31 12:11:53.747 | Loaded extension: os-console-output
2013-05-31 12:11:53.747 | Loaded extension: os-security-groups
2013-05-31 12:11:53.747 | Loaded extension: os-evacuate
2013-05-31 12:11:53.747 | Loaded extension: os-used-limits
2013-05-31 12:11:53.747 | Loaded extension: OS-FLV-EXT-DATA
2013-05-31 12:11:53.747 | Loaded extension: OS-EXT-IMG-SIZE
2013-05-31 12:11:53.747 | Loaded extension: os-server-start-stop
2013-05-31 12:11:53.747 | Loaded extension: os-cloudpipe-update
2013-05-31 12:11:53.747 | Loaded extension: OS-DCF
2013-05-31 12:11:53.747 | Loaded extension: os-admin-actions
2013-05-31 12:11:53.747 | Loaded extension: os-attach-interfaces
2013-05-31 12:11:53.747 | Loaded extension: os-fixed-ips
2013-05-31 12:11:53.747 | Loaded extension: os-instance_usage_audit_log
2013-05-31 12:11:53.748 | Loaded extension: os-consoles
2013-05-31 12:11:53.748 | Loaded extension: os-flavor-access
2013-05-31 12:11:53.748 | Loaded extension: os-baremetal-nodes
2013-05-31 12:11:53.748 | osapi_compute listening on 127.0.0.1:40408
2013-05-31 12:11:53.748 | http://127.0.0.1:40408/v2
2013-05-31 12:11:53.748 | Starting conductor node (version 2013.2)
2013-05-31 12:11:53.748 | (31622) wsgi starting up on http://127.0.0.1:40408/
2013-05-31 12:11:53.748 |
2013-05-31 12:11:53.748 | Doing GET on /v2
2013-05-31 12:11:53.748 | (31622) accepted ('127.0.0.1', 60809)
2013-05-31 12:11:53.748 |
2013-05-31 12:11:53.748 | 127.0.0.1 "GET /v2 HTTP/1.1" status: 204 len: 216 time: 0.0006292
2013-05-31 12:11:53.748 |
2013-05-31 12:11:53.748 | Doing GET on /v2/openstack//images/detail
2013-05-31 12:11:53.748 | (31622) accepted ('127.0.0.1', 60810)
2013-05-31 12:11:53.748 |
2013-05-31 12:11:53.749 | 127.0.0.1 "GET /v2/openstack//images/detail HTTP/1.1" status: 200 len: 5251 time: 0.8105230
2013-05-31 12:11:53.749 |
2013-05-31 12:11:53.749 | Doing GET on /v2/openstack//flavors/detail
2013-05-31 12:11:53.749 | (31622) accepted ('127.0.0.1', 60813)
2013-05-31 12:11:53.749 |
2013-05-31 12:11:53.749 | 127.0.0.1 "GET /v2/openstack//flavors/detail HTTP/1.1" status: 200 len: 2013 time: 0.0107400
2013-05-31 12:11:53.749 |
2013-05-31 12:11:53.749 | Doing GET on /v2/openstack//servers/detail
2013-05-31 12:11:53.749 | (31622) accepted ('127.0.0.1', 60814)
2013-05-31 12:11:53.749 |
2013-05-31 12:11:53.749 | 127.0.0.1 "GET /v2/openstack//servers/detail HTTP/1.1" status: 200 len: 187 time: 0.0314510
2013-05-31 12:11:53.749 |
2013-05-31 12:11:53.749 | Doing POST on /v2/openstack//servers
2013-05-31 12:11:53.749 | Body: {"server": {"flavorRef": "http://fake.server/1";, "name": "serverGNYOHCDP", "imageRef": "http://fake.server/70a599e0-31e7-49b7-b260-868f441e862b"}}
2013-05-31 12:11:53.749 | (31622) accepted ('127.0.0.1', 60815)
2013-05-31 12:11:53.750 |
2013-05-31 12:11:53.750 | Starting instance...
2013-05-31 12:11:53.750 | Attempting claim: memory 512 MB, disk 1 GB, VCPUs 1
2013-05-31 12:11:53.750 | Total Memory: 8192 MB, used: 512.00 MB
2013-05-31 12:11:53.750 | Memory limit not specified, defaulting to unlimited
2013-05-31 12:11:53.750 | Total Disk: 1028 GB, used: 0.00 GB
2013-05-31 12:11:53.750 | Disk limit not specified, defaulting to unlimited
2013-05-31 12:11:53.750 | Total CPU: 1 VCPUs, used: 0.00 VCPUs
2013-05-31 12:11:53.750 | CPU limit not specified, defaulting to unlimited
2013-05-31 12:11:53.750 | Claim successful
2013-05-31 12:11:53.750 | 127.0.0.1 "POST /v2/openstack//servers HTTP/1.1" status: 202 len: 659 time: 0.5113389
2013-05-31 12:13:26.360 |
2013-05-31 12:13:26.360 | Doing GET on /v2/openstack//servers/42e6f3a9-6618-4197-97d4-67c7dbe33bfd
2013-05-31 12:13:26.360 | (31622) accepted ('127.0.0.1', 60817)
2013-05-31 12:13:26.360 |
2013-05-31 12:13:26.360 | 127.0.0.1 "GET /v2/openstack//servers/42e6f3a9-6618-4197-97d4-67c7dbe33bfd HTTP/1.1" status: 200 len: 1470 time: 0.0540881
2013-05-31 12:13:26.360 |
2013-05-31 12:13:26.360 | Doing DELETE on /v2/openstack//servers/42e6f3a9-6618-4197-97d4-67c7dbe33bfd
2013-05-31 12:13:26.361 | (31622) accepted ('127.0.0.1', 60818)
2013-05-31 12:13:26.361 |
2013-05-31 12:13:26.361 | Auditing locally available compute resources
2013-05-31 12:13:26.361 | 127.0.0.1 "DELETE /v2/openstack//servers/42e6f3a9-6618-4197-97d4-67c7dbe33bfd HTTP/1.1" status: 204 len: 179 time: 0.2205250
2013-05-31 12:13:26.361 |
2013-05-31 12:13:26.361 | Doing GET on /v2/openstack//servers/42e6f3a9-6618-4197-97d4-67c7dbe33bfd
2013-05-31 12:13:26.361 | (31622) accepted ('127.0.0.1', 60821)
2013-05-31 12:13:26.361 |
2013-05-31 12:13:26.361 | 127.0.0.1 "GET /v2/openstack//servers/42e6f3a9-6618-4197-97d4-67c7dbe33bfd HTTP/1.1" status: 200 len: 1461 time: 0.0791640
2013-05-31 12:13:26.361 |
2013-05-31 12:13:26.361 | Doing POST on /v2/openstack//servers/42e6f3a9-6618-4197-97d4-67c7dbe33bfd/action
2013-05-31 12:13:26.361 | Body: {"restore": {}}
2013-05-31 12:13:26.361 | Free ram (MB): 7168
2013-05-31 12:13:26.361 | Free disk (GB): 1027
2013-05-31 12:13:26.362 | Free VCPUS: 0
2013-05-31 12:13:26.362 | (31622) accepted ('127.0.0.1', 60822)
2013-05-31 12:13:26.362 |
2013-05-31 12:13:26.362 | Compute_service record updated for 760497b8a7e84358b92c6bbe81556192:fake-mini
2013-05-31 12:13:26.362 | Reclaiming deleted instance
2013-05-31 12:13:26.362 | Terminating instance
2013-05-31 12:13:26.362 | 127.0.0.1 "POST /v2/openstack//servers/42e6f3a9-6618-4197-97d4-67c7dbe33bfd/action HTTP/1.1" status: 202 len: 121 time: 0.2906492
2013-05-31 12:13:26.362 |
2013-05-31 12:13:26.362 | Doing GET on /v2/openstack//servers/42e6f3a9-6618-4197-97d4-67c7dbe33bfd
2013-05-31 12:13:26.362 | (31622) accepted ('127.0.0.1', 60823)
2013-05-31 12:13:26.362 |
2013-05-31 12:13:26.362 | 127.0.0.1 "GET /v2/openstack//servers/42e6f3a9-6618-4197-97d4-67c7dbe33bfd HTTP/1.1" status: 200 len: 1457 time: 0.0971241
2013-05-31 12:13:26.362 |
2013-05-31 12:13:26.362 | Updating bandwidth usage cache
2013-05-31 12:13:26.362 | Doing GET on /v2/openstack//servers/42e6f3a9-6618-4197-97d4-67c7dbe33bfd
2013-05-31 12:13:26.362 | (31622) accepted ('127.0.0.1', 60825)
2013-05-31 12:13:26.363 |
2013-05-31 12:13:26.363 | HTTP exception thrown: Instance could not be found
2013-05-31 12:13:26.363 | 127.0.0.1 "GET /v2/openstack//servers/42e6f3a9-6618-4197-97d4-67c7dbe33bfd HTTP/1.1" status: 404 len: 267 time: 0.0434902
2013-05-31 12:13:26.363 |
2013-05-31 12:13:26.363 | Stopping WSGI server.
2013-05-31 12:13:26.363 | }}}
2013-05-31 12:13:26.363 |
2013-05-31 12:13:26.363 | Traceback (most recent call last):
2013-05-31 12:13:26.363 | File "/home/jenkins/workspace/gate-nova-python27/nova/tests/integrated/test_servers.py", line 216, in test_deferred_delete_restore
2013-05-31 12:13:26.363 | found_server = self._wait_for_state_change(found_server, 'DELETED')
2013-05-31 12:13:26.363 | File "/home/jenkins/workspace/gate-nova-python27/nova/tests/integrated/test_servers.py", line 39, in _wait_for_state_change
2013-05-31 12:13:26.363 | server = self.api.get_server(server['id'])
2013-05-31 12:13:26.363 | File "/home/jenkins/workspace/gate-nova-python27/nova/tests/integrated/api/client.py", line 211, in get_server
2013-05-31 12:13:26.363 | return self.api_get('/servers/%s' % server_id)['server']
2013-05-31 12:13:26.363 | File "/home/jenkins/workspace/gate-nova-python27/nova/tests/integrated/api/client.py", line 180, in api_get
2013-05-31 12:13:26.364 | response = self.api_request(relative_uri, **kwargs)
2013-05-31 12:13:26.364 | File "/home/jenkins/workspace/gate-nova-python27/nova/tests/integrated/api/client.py", line 160, in api_request
2013-05-31 12:13:26.364 | raise OpenStackApiNotFoundException(response=response)
2013-05-31 12:13:26.364 | OpenStackApiNotFoundException: Item not found
2013-05-31 12:13:26.364 | Status Code: 404
2013-05-31 12:13:26.364 | Body: {"itemNotFound": {"message": "Instance could not be found", "code": 404}}
2013-05-31 12:13:26.364 | ======================================================================
2013-05-31 12:13:26.364 | FAIL: process-returncode
2013-05-31 12:13:26.364 | tags: worker-3
2013-05-31 12:13:26.364 | ----------------------------------------------------------------------
2013-05-31 12:13:26.364 | Binary content:
2013-05-31 12:13:26.364 | traceback (test/plain; charset="utf8")
2013-05-31 12:13:26.364 | Ran 6154 tests in 322.356s
2013-05-31 12:13:26.364 | FAILED (id=0, failures=2, skips=8)
2013-05-31 12:13:26.364 | error: testr failed (1)
2013-05-31 12:13:26.581 | ERROR: InvocationError: '/home/jenkins/workspace/gate-nova-python27/.tox/py27/bin/python setup.py testr --slowest --testr-args='
2013-05-31 12:13:26.582 | ___________________________________ summary ____________________________________
2013-05-31 12:13:26.582 | ERROR: py27: commands failed
2013-05-31 12:13:26.600 | + result=1
2013-05-31 12:13:26.600 | + echo 'Begin pip freeze output from test virtualenv:'
2013-05-31 12:13:26.600 | Begin pip freeze output from test virtualenv:
2013-05-31 12:13:26.600 | + echo ======================================================================
2013-05-31 12:13:26.600 | ======================================================================
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1186243/+subscriptions
