yahoo-eng-team team mailing list archive

[OpenStack Infra <1978444@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.opendev.org/c/openstack/nova/+/845543
Committed: https://opendev.org/openstack/nova/commit/8f4b740ca5292556f8e953a30f2a11ed4fbc2945
Submitter: "Zuul (22348)"
Branch:    master

commit 8f4b740ca5292556f8e953a30f2a11ed4fbc2945
Author: Takashi Kajinami <tkajinam@xxxxxxxxxx>
Date:   Mon Jun 13 14:48:24 2022 +0900

    Retry attachment delete API call for 504 Gateway Timeout
    
    When cinder-api runs behind a load balancer(eg haproxy), the load
    balancer can return 504 Gateway Timeout when cinder-api does not
    respond within timeout. This change ensures nova retries deleting
    a volume attachment in that case.
    
    Also this change makes nova ignore 404 in the API call. This is
    required because cinder might continue deleting the attachment even if
    the load balancer returns 504. This also helps us in the situation
    where the volume attachment was accidentally removed by users.
    
    Closes-Bug: #1978444
    Change-Id: I593011d9f4c43cdae7a3d53b556c6e2a2b939989


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1978444

Title:
  Volume can't be detached if attachment delete api call fails with 504
  gateway timeout

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  Description
  ===========
  When cinder-api is running behind load balancer like haproxy, the load balancer can return 504 if it can not receive response from cinder-api within timeout.
  When this timeout occurs while detaching a volume, this results in un-detachable volume.

   - nova-compute calls delete attachment api in cinder
   - haproxy detects server timeout and returns 504
   - cinder continues processing the API and removes the attachment
   - nova-compute immediately aborts the volume detachment and leaves the bdm
   - when a client tries to detach the volume again, the detachment fails because the attachment no longer exists in Nova

  See for details https://bugzilla.redhat.com/show_bug.cgi?id=2002643

  Steps to reproduce
  ==================
  * Stop cinder-volume
  * Detach a volume from an instance
  * Start cinder-volume
  * Detach the volume again

  Expected result
  ===============
  * Volume can be detached after cinder-volume is recovered

  Actual result
  ===============
  * Volume can't be detached

  Environment
  ===========
  * The issue was initially found in stable/train

  Logs & Configs
  ==============
  * See https://bugzilla.redhat.com/show_bug.cgi?id=2002643#c1

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1978444/+subscriptions