yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1812969] Re: catch VolumeAttachmentNotFound when attach failed

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: OpenStack Infra <1812969@xxxxxxxxxxxxxxxxxx>
Date: Thu, 14 Feb 2019 08:01:18 -0000
Reply-to: Bug 1812969 <1812969@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Reviewed:  https://review.openstack.org/632657
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=595866eb62db56fbd4f965619031ac902f99441a
Submitter: Zuul
Branch:    master

commit 595866eb62db56fbd4f965619031ac902f99441a
Author: yenai <yenai@xxxxxxxxxxxxxxxxxxxx>
Date:   Wed Jan 23 13:54:47 2019 +0800

    Ignore VolumeAttachmentNotFound exception in compute.manager
    
    DriverVolumeBlockDevice will delete volume attachment when attach
    fails, codes link:
    https://github.com/openstack/nova/blob/907c7d2cf/nova/virt/block_device.py#L561-L568
    However, nova.compute.manager will delete it again and it will raise
    VolumeAttachmentNotFound exception. This outputs an incorrect error
    log and this exception should be ignored.
    
    Change-Id: I939c09e5b0efb3b17a9855af227e6d60c64d23e2
    Closes-Bug: #1812969


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1812969

Title:
  catch VolumeAttachmentNotFound when attach failed

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) queens series:
  Confirmed
Status in OpenStack Compute (nova) rocky series:
  Confirmed

Bug description:
  [stack@localhost devstack]$ cinder type-create encrypt-NoOpEncryptor
  +--------------------------------------+-----------------------+-------------+-----------+
  | ID | Name | Description | Is_Public |
  +--------------------------------------+-----------------------+-------------+-----------+
  | 054fad16-6b6b-4426-ac9b-af63bc43d113 | encrypt-NoOpEncryptor | - | True |
  +--------------------------------------+-----------------------+-------------+-----------+
  [stack@localhost devstack]$ cinder encryption-type-create 054fad16-6b6b-4426-ac9b-af63bc43d113 --control-location front-end NoOpEncryptor
  +--------------------------------------+---------------+--------+----------+------------------+
  | Volume Type ID | Provider | Cipher | Key Size | Control Location |
  +--------------------------------------+---------------+--------+----------+------------------+
  | 054fad16-6b6b-4426-ac9b-af63bc43d113 | NoOpEncryptor | - | - | front-end |
  +--------------------------------------+---------------+--------+----------+------------------+
  [stack@localhost devstack]$ cinder create --volume-type encrypt-NoOpEncryptor --name yenai 1
  +--------------------------------+--------------------------------------+
  | Property | Value |
  +--------------------------------+--------------------------------------+
  | attachments | [] |
  | availability_zone | nova |
  | bootable | false |
  | consistencygroup_id | None |
  | created_at | 2019-01-12T01:36:36.000000 |
  | description | None |
  | encrypted | True |
  | id | 247bd974-3cb5-44a5-9b7d-599ca2fc9bda |
  | metadata | {} |
  | migration_status | None |
  | multiattach | False |
  | name | yenai |
  | os-vol-host-attr:host | None |
  | os-vol-mig-status-attr:migstat | None |
  | os-vol-mig-status-attr:name_id | None |
  | os-vol-tenant-attr:tenant_id | 9b881cf6b92049b8a155178274662836 |
  | replication_status | None |
  | size | 1 |
  | snapshot_id | None |
  | source_volid | None |
  | status | creating |
  | updated_at | None |
  | user_id | 657205033877433db3dd8dadb212d9fb |
  | volume_type | encrypt-NoOpEncryptor |
  +--------------------------------+--------------------------------------+
  [stack@localhost libvirt]$ cinder list
  +--------------------------------------+-----------+-------+------+-----------------------+----------+--------------------------------------+
  | ID | Status | Name | Size | Volume Type | Bootable | Attached to |
  +--------------------------------------+-----------+-------+------+-----------------------+----------+--------------------------------------+
  | 247bd974-3cb5-44a5-9b7d-599ca2fc9bda | available | yenai | 1 | encrypt-NoOpEncryptor | false | |
  | f1c15752-c247-470b-9a5f-1f5559f51ce7 | in-use | yenai | 1 | encrypt-luks | false | 6ce2254a-0e05-4e63-874e-44d112bed7d0 |
  +--------------------------------------+-----------+-------+------+-----------------------+----------+--------------------------------------+
  [[stack@localhost devstack]$ nova volume-attach 6ce2254a-0e05-4e63-874e-44d112bed7d0 247bd974-3cb5-44a5-9b7d-599ca2fc9bda
  +----------+--------------------------------------+
  | Property | Value |
  +----------+--------------------------------------+
  | device | /dev/vdc |
  | id | 247bd974-3cb5-44a5-9b7d-599ca2fc9bda |
  | serverId | 6ce2254a-0e05-4e63-874e-44d112bed7d0 |
  | volumeId | 247bd974-3cb5-44a5-9b7d-599ca2fc9bda |
  +----------+--------------------------------------+
  [stack@localhost devstack]$
  Jan 12 13:58:48 localhost.localdomain nova-compute[95555]: ERROR nova.virt.block_device [None req-c7c9bfab-75a6-41b8-a677-d00aac80b11b admin admin] [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] Driver failed to attach volume 247bd974-3cb5-44a5-9b7d-599ca2fc9bda at /dev/vdc: TypeError: attach_volume() got an unexpected keyword argument 'cipher'
  Jan 12 13:58:48 localhost.localdomain nova-compute[95555]: ERROR nova.virt.block_device [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] Traceback (most recent call last):
  Jan 12 13:58:48 localhost.localdomain nova-compute[95555]: ERROR nova.virt.block_device [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0]   File "/opt/stack/nova/nova/virt/block_device.py", line 559, in _volume_attach
  Jan 12 13:58:48 localhost.localdomain nova-compute[95555]: ERROR nova.virt.block_device [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0]     device_type=self['device_type'], encryption=encryption)
  Jan 12 13:58:48 localhost.localdomain nova-compute[95555]: ERROR nova.virt.block_device [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1464, in attach_volume
  Jan 12 13:58:48 localhost.localdomain nova-compute[95555]: ERROR nova.virt.block_device [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0]     encryption=encryption)
  Jan 12 13:58:48 localhost.localdomain nova-compute[95555]: ERROR nova.virt.block_device [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1264, in _connect_volume
  Jan 12 13:58:48 localhost.localdomain nova-compute[95555]: ERROR nova.virt.block_device [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0]     vol_driver.disconnect_volume(connection_info, instance)
  Jan 12 13:58:48 localhost.localdomain nova-compute[95555]: ERROR nova.virt.block_device [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  Jan 12 13:58:48 localhost.localdomain nova-compute[95555]: ERROR nova.virt.block_device [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0]     self.force_reraise()
  Jan 12 13:58:48 localhost.localdomain nova-compute[95555]: ERROR nova.virt.block_device [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Jan 12 13:58:48 localhost.localdomain nova-compute[95555]: ERROR nova.virt.block_device [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0]     six.reraise(self.type_, self.value, self.tb)
  Jan 12 13:58:48 localhost.localdomain nova-compute[95555]: ERROR nova.virt.block_device [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1261, in _connect_volume
  Jan 12 13:58:48 localhost.localdomain nova-compute[95555]: ERROR nova.virt.block_device [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0]     context, connection_info, encryption, allow_native_luks)
  Jan 12 13:58:48 localhost.localdomain nova-compute[95555]: ERROR nova.virt.block_device [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1399, in _attach_encryptor
  Jan 12 13:58:48 localhost.localdomain nova-compute[95555]: ERROR nova.virt.block_device [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0]     encryptor.attach_volume(context, **encryption)
  Jan 12 13:58:48 localhost.localdomain nova-compute[95555]: ERROR nova.virt.block_device [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] TypeError: attach_volume() got an unexpected keyword argument 'cipher'
  Jan 12 13:58:48 localhost.localdomain nova-compute[95555]: ERROR nova.virt.block_device [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0]
  Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [None req-c7c9bfab-75a6-41b8-a677-d00aac80b11b admin admin] [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] Failed to attach 247bd974-3cb5-44a5-9b7d-599ca2fc9bda at /dev/vdc: TypeError: attach_volume() got an unexpected keyword argument 'cipher'
  Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] Traceback (most recent call last):
  Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0]   File "/opt/stack/nova/nova/compute/manager.py", line 5442, in _attach_volume
  Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0]     do_driver_attach=True)
  Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0]   File "/opt/stack/nova/nova/virt/block_device.py", line 46, in wrapped
  Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0]     ret_val = method(obj, context, *args, **kwargs)
  Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0]   File "/opt/stack/nova/nova/virt/block_device.py", line 643, in attach
  Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0]     virt_driver, do_driver_attach)
  Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0]   File "/opt/stack/nova/nova/virt/block_device.py", line 621, in _do_attach
  Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0]     do_driver_attach)
  Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0]   File "/opt/stack/nova/nova/virt/block_device.py", line 568, in _volume_attach
  Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0]     attachment_id)
  Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0]     self.force_reraise()
  Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0]     six.reraise(self.type_, self.value, self.tb)
  Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0]   File "/opt/stack/nova/nova/virt/block_device.py", line 559, in _volume_attach
  Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0]     device_type=self['device_type'], encryption=encryption)
  Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1464, in attach_volume
  Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0]     encryption=encryption)
  Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1264, in _connect_volume
  Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0]     vol_driver.disconnect_volume(connection_info, instance)
  Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0]     self.force_reraise()
  Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0]     six.reraise(self.type_, self.value, self.tb)
  Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1261, in _connect_volume
  Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0]     context, connection_info, encryption, allow_native_luks)
  Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1399, in _attach_encryptor
  Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0]     encryptor.attach_volume(context, **encryption)
  Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] TypeError: attach_volume() got an unexpected keyword argument 'cipher'
  Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0]
  Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.volume.cinder [None req-c7c9bfab-75a6-41b8-a677-d00aac80b11b admin admin] Delete attachment failed for attachment 0a85e2d1-4e33-4f9e-ac87-f5c8f0aaec78. Error: Volume attachment could not be found with filter: attachment_id = 0a85e2d1-4e33-4f9e-ac87-f5c8f0aaec78. (HTTP 404) (Request-ID: req-97e95f31-4dcb-4714-bcea-0c80508e2b08) Code: 404: NotFound: Volume attachment could not be found with filter: attachment_id = 0a85e2d1-4e33-4f9e-ac87-f5c8f0aaec78. (HTTP 404) (Request-ID: req-97e95f31-4dcb-4714-bcea-0c80508e2b08)

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1812969/+subscriptions
References

[Bug 1812969] [NEW] catch VolumeAttachmentNotFound when attach failed
From: yenai, 2019-01-23