yahoo-eng-team team mailing list archive

Thread
Date

[Bug 1714924] Re: placement api fails when nova tries to delete resource allocation after failed evacuation

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Balazs Gibizer <balazs.gibizer@xxxxxxxxxxxx>
Date: Wed, 27 Jun 2018 13:25:09 -0000
Reply-to: Bug 1714924 <1714924@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx

At least the test test_evacuate_with_no_compute that showed this stack
trace does not produce the same trace any more. So I think the evacuate
part also OK on master.

** Changed in: nova
       Status: Confirmed => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1714924

Title:
  placement api fails when nova tries to delete resource allocation
  after failed evacuation

Status in OpenStack Compute (nova):
  Invalid

Bug description:
  During the investigation of bug #1713783 (After failed evacuation the
  recovered source compute tries to delete the instance) we noticed that
  nova tries to PUT an empty '{}' allocations for an instance and that
  fails on the placement side. As bug #1713783 now seems to be solved by
  keeping the existing behavior, i.e. deleting the instance after failed
  evacuation. We have to fix nova not to try to PUT the allocation but
  actually DELETE the allocation of the instance in placement.

  The problem can be reproduced with the regression test proposed in
  https://review.openstack.org/#/c/498482/ which produce the following
  stack trace:

      2017-09-04 11:33:24,989 INFO [nova.service] Starting compute node (version 16.0.1)
      2017-09-04 11:33:25,005 INFO [nova.compute.manager] Deleting instance as it has been evacuated from this host
      2017-09-04 11:33:25,060 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/allocations/4d056cc0-b227-4f79-baab-f3e3cd1a6d00" status: 200 len: 134 microversion: 1.0
      2017-09-04 11:33:25,066 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "PUT /placement/allocations/4d056cc0-b227-4f79-baab-f3e3cd1a6d00" status: 400 len: 651 microversion: 1.10
      2017-09-04 11:33:25,067 WARNING [nova.scheduler.client.report] Failed to save allocation for 4d056cc0-b227-4f79-baab-f3e3cd1a6d00. Got HTTP 400: <html>
       <head>
        <title>400 Bad Request</title>
       </head>
       <body>
        <h1>400 Bad Request</h1>
        The server could not comply with the request since it is either malformed or otherwise incorrect.<br /><br />
      JSON does not validate: {} does not have enough properties
      
      Failed validating 'minProperties' in schema['properties']['allocations']['items']['properties']['resources']:
          {'additionalProperties': False,
           'minProperties': 1,
           'patternProperties': {'^[0-9A-Z_]+$': {'minimum': 1,
                                                  'type': 'integer'}},
           'type': 'object'}
      
      On instance['allocations'][0]['resources']:
          {}
      
      
       </body>
      </html>
      2017-09-04 11:33:25,067 ERROR [nova.compute.resource_tracker] Failed to clean allocation of evacuated instance on the source node  bc78aa7f-07df-4e06-bb77-71aec7d92e5c

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1714924/+subscriptions

References

[Bug 1714924] [NEW] placement api fails when nova tries to delete resource allocation after failed evacuation
From: Balazs Gibizer, 2017-09-04